Probability

Blaise Pascal and Pierre Fermat developed the fundamental principles of probability in a series of letters exchanged starting in the year 1654.

Definitions

Experiment is the process of obtaining an observation. For example: Toss a coin twice and observe what happens.

Outcome is the possible result of the experiment. All possible outcomes of the experiment are called the sample space. The experiment of tossing a coin twice results in one of four possible outcomes - Tail/Tail, Head/Head, Head/Tail, or Tail/Head.

Event is a set of outcomes of the experiment. One event would be: Observe at least one head. This event consists of the three outcomes HH, HT, TH. A compound event is an event composed of two or more separate events.

Independent events — two events A and B are independent if knowing whether one occurred does not change the probability of the other. For example, let event A be observing a head when flipping a coin, and event B be observing a tail when flipping another coin. Assuming the flips are independent, the first result tells us nothing about the second.

Mutually exclusive events - two events are mutually exclusive if they cannot happen at the same time i.e. they have no outcomes in common. A single coin is tossed. There are two events: observing a head or observing a tail. Observing a head excludes the possibility of observing a tail. Two events are non-mutually exclusive if they have one or more outcomes in common. A single die is tossed. Event A: observe a four. Event B: observe an even number. These events have one outcome in common since an "even number" consists of the numbers 2, 4, and 6.

Probability — a number between 0 and 1 that measures how likely an event is. A probability of 1 means the event occurs almost surely; a probability of 0 means it occurs with zero probability. In a finite sample space with positive probability for every outcome, these correspond to certainty and impossibility. Continuous models require care: an exact value can be possible while having zero probability.

Arithmetic mean of a set of outcomes is usually called the average value of these outcomes. To find the mean of the numbers 1,8,6,4,7, we add the numbers together-26-and divide by 5 = 5.2.

Variability shows how concentrated or spread out the outcomes are around the arithmetic mean.

Expectation is the average value we expect to observe if we perform a large number of experiments. Also called expected value - the probability-weighted sum of all possible outcomes.

Population - the total number of something - outcomes, objects, events, etc. It is a group that has at least one characteristic in common from which our sample of data is selected.

Sample — a subset of a population studied to learn about that population. A larger random sample generally gives a more precise estimate, but size does not remove selection bias. For a large population, the absolute sample size often matters more than the fraction sampled. A random sample of 3,000 from the entire U.S. is usually more useful for estimating a national proportion than a sample of 40 from a particular university. In simple random sampling, every subset of the chosen size has the same chance of selection.

How do we decide the probability of an event?

Probability models describe uncertainty and, under suitable assumptions, the frequencies we expect over many trials. They can support predictions about averages without determining what will happen in any particular trial.

There are three ways to measure probability: the logical way, the relative frequency way, and the subjective way.

The logical way

The logical way can be used in situations where we know the number of outcomes and where all these outcomes are equally likely. For example in games of chance, we find the probability by dividing the number of outcomes that are favorable to the event (yield the outcome we look for) with the total number of possible outcomes. Observe that this definition can only be applied if we can analyze a situation into equally likely outcomes.

What is the probability that we observe one head when tossing a coin? The number of equally likely favorable outcomes is one (head) and the total number of possible equally likely outcomes is two (head and tail) and therefore the probability is ½ or 50%.

Relative frequency

When an experiment can be repeated under stable conditions, we can estimate an event’s probability from the proportion of trials in which it occurs. We can run experiments or use representative observations of past events. Knowing the possible outcomes alone is not enough; we also need to know how likely they are.

By representative we mean that the information must be based on the relative frequency of historical data over a large number of independent experiments or observations of the same reference class under essentially the same conditions. The reference class is the one for which the distribution of outcomes is known or can be reasonably estimated. The more relevant cases we examine, the better our chance is to estimate correctly the underlying probability.

Conduct an experiment to estimate how likely a coin is to land heads. Toss it 1000 times. If heads appears 400 times, the relative frequency is 400/1000. Toss it 2000 times. If heads appears 900 times, the relative frequency is 900/2000. Under independent, identically distributed tosses, the relative frequency tends toward the underlying probability as the number of tosses grows. It need not move closer on every toss. For a fair coin, the limiting value is ½.

Insurance companies use relative frequencies to estimate how often losses occur, how they are distributed over time, and how large they may be. If the past is a representative guide to the future, observed frequencies of a particular "accident" can help estimate its probability.

Suppose the probability that a given house will burn during a specified period is 0.3%. This means that the insurance company found historical data and other indicators about a large number of houses (for example the reference class is "50 years of data of fires in a given area") and discovered that, in the past, 3 out of 1,000 houses in a given area burned during the period being measured. It also means that, assuming there are no changes in the causes of these fires, we can predict approximately the same proportion of fires in the future. An insurance company knows that in a given year a certain percentage of their policyholders will have an accident. They don't know which ones, but they diversify their risk by insuring many individuals. What is unpredictable for one person can be predictable for a large population. But they must make sure the events are independent and that not a single event or a confluence of independent events affects multiple policyholders causing the insurer to pay out on many claims at one time. For example, an insurance company who provides fire insurance to a number of buildings in a single block may face ruin if a large fire happens.

Subjective probability

If an experiment is not repeatable or when there is no representative historical relative frequency, or comparable data, the probability is then a measure of our personal degree of belief in the likelihood of an event happening. We have to make a subjective evaluation or a personal estimation using whatever information is available. But we can't just assign any number to events. They must agree with the rules of probability.

A New York Rangers supporter might say, "I believe that the NY Rangers have a 90% chance of winning their next match since they have been playing so well."

Rules of probability

When two events are independent (no event can influence the probability of the other), the probability that they both will happen is the product of their individual probabilities. We can write this as: Both A and B will happen = P(A) x P(B).

A company has two independent manufacturing processes. In process one, 5% of the produced items are defective, and in process two, 3%. If we pick one item from each process, how likely is it that both items are defective? 0.15% (0.05 x 0.03).

For dependent events, multiply the probability of the first by the conditional probability of the second given the first. Suppose we roll a fair die. Let event A be observing an even number, and event B be observing a number less than 4. Given B, the probability of A is 1/3. A conditional probability describes an event’s probability given that another event occurred; it is defined for independent events too. Here the probability of A given B is 1/3 because B restricts the possible results to 1,2 or 3, and only 2 is even.

What is the probability of two boys in a two-child family, given that there is at least one boy? Assume independent births, equally likely boy/girl outcomes, and that this information was obtained without favoring a particular child or family. The equally likely ordered outcomes are Boy/Boy, Girl/Girl, Boy/Girl, and Girl/Boy. Knowing there is "at least one boy" rules out "Girl/Girl". The probability is therefore 1/3, or about 33%.

What is the probability that there are two boys in a two-child family given that the first-born is a boy? The number of outcomes that are equally likely to happen are: Boy/Boy - Girl/Girl - Boy/Girl - Girl/Boy. Since we already know that the older child is a boy, we can rule out scenario GB and GG. The probability is 50%.

A problem in conditional probability that has caused many mathematics professors problems, is the Monty Hall Dilemma. The columnist Marilyn vos Savant (Parade 1990, Sept 9, p. 13.) asked the following problem:

"Suppose you're on a game show, and you're given the choice of three doors. Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, 'Do you want to pick door No. 2?' Is it to your advantage to switch your choice?"

How would you answer? Assume the car is equally likely to be behind each door, the host knows its location, always opens an unchosen goat door, and always offers a switch. Make a table of possible outcomes and count how often switching wins.

Door 1	Door 2	Door 3
Car	Goat	Goat
Goat	Car	Goat
Goat	Goat	Car

Assume you choose door number 1. What are then the consequences if the car is in door 1, 2 or 3?

Car behind door	Host opens door	You switch	You don't switch
1	2	Lose	Win
2	3	Win	Lose
3	2	Win	Lose
		2/3	1/3

We should always switch doors since we win 2/3 of the time. The key to this problem is that we know ahead of the game (conditional) that the host knows what is behind each door and always opens a door with a goat behind it.

When two events are mutually exclusive (can't both happen at the same time), the probability that one or the other will happen is the sum of their respective probabilities. We can write this as: Either A or B will happen = P(A) + P(B).

What is the probability of getting either a two or a four on one roll of a fair die? There are 6 equally likely outcomes, and the events ("getting a two" and "getting a four") cannot both occur on the same roll. Each has probability one out of six. The probability of getting a two or a four is 1/6 + 1/6, or about 33%.

When two events are non-mutually exclusive (both can happen at the same time), the probability that at least one of them will happen is equal to the sum of the probabilities of the two events minus the probability that both events happen. We can write this complementary rule as: P(A) + P(B) - P(A and B).

Assume that a randomly selected Los Angeles teenager has a 25% probability of owning a surfboard, an 85% probability of owning a bicycle, and a 20% probability of owning both. The probability of owning a surfboard or a bicycle is (0.25 + 0.85) - 0.20 = 90%. Subtracting the overlap prevents counting teenagers who own both twice.

Sometimes it is easier to deal with problems if we turn them backwards. The probability of an event not happening is 1 minus the probability it will happen. If the probability of an event A is 30%, then the probability that the event won't happen is 70% because "not event A" is the complement of the event A. The sum of probability of an event happening and probability of the same event not happening is always 1.

What is the probability that we get at least one six in four rolls of a single die? We turn the question around and calculate the probability of "not getting any sixes in four rolls of a die". There are four independent events - not getting a six on the first throw... second... The probability of each one is 5/6 since there are five outcomes (1,2,3,4,5) which results in the event "no six" and each is independent of what happened before. This means that the probability of not getting any sixes is 5/6 x 5/6 x 5/6 x 5/6 or 48.2%. Therefore "the probability of getting at least one six" is 1 - 0.482 or 51.8%.

Counting possible outcomes

The multiplication principle says that if a first choice has "n" possibilities and each of those allows "m" second choices, there are nm ordered pairs of choices.

Suppose there are 4 different flights between Los Angeles and New York, 3 between New York and Boston and 5 between Boston and Bermuda. The number of itineraries assuming we can connect between any of the stated flights are 4 x 3 x 5 = 60.

Permutations or rearrangements mean the different ways we can order or arrange a number of objects.

We have 3 hats to choose from - one black, one white and one brown. In how many ways can we arrange them if the order white, black and brown is different from the order black, white and brown? This is the same as asking how many permutations there are with three hats, taken three at a time. We can arrange the hats in 6 ways: Black-White-Brown, Black-Brown-White, White-Black-Brown, White-Brown-Black, Brown-White-Black, Brown-Black-White.

Another way to think about this: We have three boxes in a row where we put a different hat in each box. We can fill the first box in three ways, since we can choose between all three hats. We can then fill the second box in two ways, since we now can choose between only two hats. We can fill the third box in only one way, since we have only one hat left. This means we can fill the box in 3 x 2 x 1 = 6 ways.

Another way to write this is 3! If we have n (6) boxes and can choose from all of them, there are n (6) choices. Then we are left with n-1 (5) choices for box number two, n-2 (4) choices for box number three and so on. The number of permutations of n boxes is n!. What n! - Factorial - means is the product of all numbers from 1 to n.

Suppose we have 12 dinner guests and distinct seats around a table. The first guest has twelve seats to choose from, the second has eleven, and so on. There are 12!, or 479,001,600, assignments of guests to those seats. This treats rotations as different seat assignments.

The number of ways we can arrange r objects from a group of n objects is called a permutation of n objects taken r at a time and is defined as n! / (n-r)!

A safe has 100 possible dial settings. To open it, a burglar must enter the correct sequence of 3 different settings. The number of ordered sequences of 3 settings chosen from 100 is 970,200 (100!/(100-3)!). If trying each sequence takes 5 seconds, testing them all would take about 56 days at a 24-hour working day.

Combinations means the different ways we can choose a number of different objects from a group of objects where no order is involved, just the number of ways of choosing them.

In how many ways can we combine 2 flavors of ice cream if we can choose from strawberry (S), vanilla (V), and chocolate (C) without repeated flavors? We can combine them in 3 ways: SV, SC, VC. VS and SV are a combination of the same ice creams. The order doesn't matter. Vanilla on the top is the same as vanilla on the bottom.

The number of ways we can select r objects from a group of n objects is called a combination of n objects taken r at a time and is defined as n! / [r!(n-r)!]

The number of ways we can select 3 people taken from a group of 10 people is 120 (10!/3!(10-3)!).

The binomial distribution

Suppose we take a true-false test with 10 questions. We don't know anything about the subject. All we can do is guess. First, we ask about getting exactly 5 questions correct. Are we likely to do that by guessing?

How should we reason? Ask: How many equally likely independent outcomes are there when we guess? There are 2 possible outcomes. Either we are right or wrong. If the test only had one question, the probability that we guess the right answer is 50%. The probability that we guess the wrong answer is also 50% (1- the probability of guessing the right answer).

What is the total number of equally likely answer patterns? Each question has 2 possible answers, so 10 questions give 2^10, or 1,024, patterns. We can answer the test in 1,024 different ways. Only one pattern gets all 10 questions right, and a different pattern gets them all wrong. The chance of getting all 10 right is 1 in 1024; the same probability applies separately to getting them all wrong. Over 1,024 randomly answered tests, we would expect about one perfect score per 1,024 attempts, and separately about one score with all 10 answers wrong. Neither count is guaranteed.

In how many ways can we be right on 5 questions? Let's go back to combinations and ask: in how many ways can we select 5 questions if we can choose from 10 questions? There are 252 ways (10!/5!(10-5)!) we can answer 10 questions to get exactly a score of 5. Since each guess has a 50% probability of being right and there are 10 questions and we want to be right on exactly 5 of them and there are 252 equally likely ways we can answer 5 questions, then the probability that we answer exactly 5 questions right is (0.5)^5 x (0.5)^5 x 252 = 24.6%

How likely is it then that we answer at least 5 questions correctly? This probability must be higher since we also can answer 6, 7, 8, 9 or 10 questions correct. Therefore we must add the probability that we guess 6, 7, 8, 9, and 10 questions correct.

In how many ways can we be right on 5 questions? 10!/5!(10-5)! = 252 ways In how many ways can we be right on 6 questions? 10!/6!(10-6)! = 210 ways In how many ways can we be right on 7 questions? 10!/7!(10-7)! = 120 ways In how many ways can we be right on 8 questions? 10!/8!(10-8)! = 45 ways In how many ways can we be right on 9 questions? 10!/9!(10-9)! = 10 ways In how many ways can we be right on 10 questions? 10!/10!(10-10)! = 1 way

Total = 638 ways

Since each guess has a 50% probability of being right and there are 10 questions and we want to be right on at least 5 of them and there are 638 equally likely ways we can answer at least 5 questions, the probability that we answer at least 5 questions right is (0.5)^5 x (0.5)^5 x 638 = 62.3%.

The example illustrated a binomial experiment. The probability distribution for a binomial experiment is: Number of possible ways of selecting k things from n things (observing k successes in n trials) x (probability of success)^k (1 - probability of success)^(n-k)

If we put in the above figures we get: 252 x (0.5)^5 x (0.5)^5 + 210 x (0.5)^6 x (0.5)^4 + 120 x (0.5)^7 x (0.5)^3 + 45 x (0.5)^8 x (0.5)^2 + 10 x (0.5)^9 x (0.5)^1 + 1 x (0.5)^10 x (0.5)^0 = 62.3%

Binomial experiments have the following characteristics: An event that is repeatable or the experiment consists of n number of identical and independent trials. There are only two outcomes in every trial - success/failure, right/wrong, present/absent, 0/1 etc. and the probabilities of success and failure are constant in every trial.

Repeated shots at a target (hit/miss), tests of a drug response (effective/not effective), or sales attempts (sale/no sale) can be modeled as binomial trials when the trials are independent and the probability of success stays constant. Those assumptions must be checked; two possible outcomes alone do not make an experiment binomial.

We toss a single die 5 times. How likely is it that you will roll exactly 3 sixes? What is success? Rolling a 6 on a single die. What's the probability of rolling a 6 on a single die? 1/6 (there are 6 outcomes and one of them is a success). What is the probability of failure? 1-1/6 = 5/6. What is the number of trials? 5. What is the number of successes out of those trials? 3. In how many ways can you roll three sixes (successes) in 5 trials? 5!/3!(5-3)! = 10. Probability = 10 x (1/6)^3 x (5/6)^2 = 3.2%

A boat has three independent engines and needs at least two to work properly. The probability that each engine works properly is 98%. The probability that all three engines work is 94.1% (0.98^3). The probability that at least one engine fails (either engine 1, 2 or 3) is therefore 5.9% (this is the same as the probability that exactly 1 engine fail + exactly 2 engines fails + exactly 3 engines fails).

What is the probability that at least 2 engines work? Let's go back to combinations and the binomial distribution: probability (3 engines work) + probability (2 engines work) = 3!/3!(3-3)! x (0.98)^3 x (0.02)^0 + 3!/2!(3-2)! x (0.98)^2 x (0.02)^1 = 99.8816%. The probability that at least two engines will fail is therefore 0.1184%. The boat will fail to work in 1 out of 845 times.

Let's add a backup engine. What is now the probability that at least 2 engines work? Probability (4 engines work) + probability (3 engines work) + probability (2 engines work) = 4!/4!(4-4)! x (0.98)^4 x (0.02)^0 + 4!/3!(4-3)! x (0.98)^3 x (0.02)^1 + 4!/2!(4-2)! x (0.98)^2 x (0.02)^2 = 99.996848%. The probability that at least three engines will fail is therefore 0.003152%. The boat will now only fail in 1 out of 31,726 times.

Note: Binomial probabilities assume independence. It may be that the failure of one engine increases the failure probability of a second engine. For example, the failure of one engine increases the load on which the second engine is run. Using one engine causes more stress and wear to the second engine, etc.

Further calculations

• Page 150: The possible number of ways we can choose 6 numbers out of 49 are 49!/(49-6)!6! = 13,983,816. 24 hours equals 1,440 minutes. One 365-day year equals 525,600 minutes. 14 million minutes equals about 27 years.
• Page 1: The probability we succeed is (0.8)^6 or 26%.
• Page 2: The probability that 10 mutually independent start-ups all succeed is 0.01% (0.4^10) but the probability that at least one succeeds is 99.4% (1 - 0.6^10).
• Page 3: The probability that at least one of the parts don't work is 86.5% (1 - 0.999^2000). Assuming independence, the probability of system failure (where at least one part must fail for the system to fail) is 1 minus the reliability of the system.
• Page 161: Assuming independence, the probability of system failure (where both navigation systems must fail for the system to fail) is the product of the probabilities of primary and back up system failing.
• Page 162: Assuming independent years and a constant annual probability, an event with one chance in 20 of happening each year has a high probability of occurring at least once over 50 years (1 - 0.95^50 = 92.3%). If there is a 5% chance that an event happens in any given year, then the chance that it won't happen is 95%. The chance that it won't happen over 50 years is 7.7%. This means that the probability that the event happens at least once is 92.3%.
• Page 162: The probability that at least one accident will happen in any given year is 3.9% (1-0.999^40). The probability that at least one accident will happen during the next 10 years is 33% (1-0.961^10).
• Page 162: The probability of a major earthquake happening in any given year (assumed to be a constant) is therefore 3.2% ((1-p)^30 = 38%). The probability that a major earthquake will happen at least once during the next 5 years is 15% (1-0.968^5).
• Page 165: In a group of 1,048,576 (2^20) people it happens to someone. In a country with 280 million people, if each person has one independent opportunity per day for an event with a one-in-a-million chance, the expected daily count is 280 (1/1,000,000 x 280 million).
• Page 165: 1 person has 365 possible birthdays if we assume that there are 365 days to choose from and that all birthdays are equally likely to happen. When there are 2 persons in a group the second person can choose among 364 possible birthdays that are not shared with the first person. The second person only shares 1 day with the first person. The chance that 2 persons share birthdays is therefore 1 out of 365 or 0.27%. When there are 3 people in a group it is easier to find the likelihood that 2 of them share a birthday by first finding out how likely it is that no one of these 3 people share birthdays. When there are 3 people in a group the third person can choose among 363 possible birthdays that are not shared with any of the first 2 persons. This means that the chance that the third person will not share birthday with any of the first 2 persons is 363 out of 365 or 99.45%.

To find the probability that multiple events happen, we multiply the individual probabilities together. The probability that no one in a group of 3, share birthday is therefore: 365/365 x 364/365 x 363/365 = 99.18%. Therefore, the probability that 2 people in a group of 3 share birthdays is 1 - 0.9918 or 0.82%. Let's repeat this procedure for a group of 23 people:

365/365 x 364/365 x 363/365 x ... x 343/365 = 49.3%.

Therefore, the probability that 2 people in a group of 23, share birthdays is 1 - 0.493 or 50.7%.

• Page 178: After 10 independent guesses per monkey, the expected number with a perfect record of predicting interest-rate direction is about one (1,000 x 0.5^10). This is an average, not a guaranteed survivor.
• Page 180: The number of ways to get 2 successes in 10 trials are 10!/(10-2)!2! or 45. The probability is 45 x (0.8)^2 x (0.2)^8 or 0.007%.