Descriptive Statistics and Deceptive Descriptions

Announcements

• Lab 0 was released on LMS, and complete setup before the next lecture on January 27.
• A Python tutorial, led by Shizza, is scheduled for Zoom.

Notebooks

• Notebooks enable the telling a narrative.
• Notebooks include code documentation, the thought process, a markdown language, and output.

Descriptive Statistics

• Descriptive stats are useful to presents meaningful information in an easily accessible way, as succinct metrics.
• Descriptive statistics can summarize careers by batting average and income distribution by the Gini Index.
• Over-reliance on descriptive statistics can lead to misleading conclusions.

Deceptive Descriptions

• Occurs when the use of statistics describes complex phenomenon not exactly.
• An example could be assesing the economic health of the middle class of a country.
• The per capita income could be analyzed over 30 years.
• For example, the average income in the US in 1980 climbed from $7,787 to $26,487 in 2010.
• Technically this statement is correct, it is wrong to question if the figures are adjusted for inflation.
• The average income in America is not equal to the income of the average america.
• It's important to compare the change in median of wages adjusted for inflation.

Printer Quality Example

• A boss shares files of data, one with warranty information of 57,334 printers sold last year.
• The other file has information on the 994,773 laser printer sold by a competetor.
• Each printer documents the number of quality problems during the warranty period.
• The average number of quality problems per printer sold by the competitor is 2.8, but 9.1 for your firm.
• However, the median number of quality problems by sold printer is 2 for the competitor, but 1 for your firm.
• Outliers inflate the mean but not the median.
• A fraction of printers have a huge number of quality complaints.

Median, Quartiles, Deciles, and Percentiles

• The median divides the distribution in hlaf.
• The median is the point where half the observations are above, and the other half are below.
• Distributions can be divided into quartiles, deciles, or percentiles.
• The first quartile is the bottom 25% of observations.
• The first decile is the bottom 10% of observations.
• The first percentile is the bottom 1% of observations.
• They describe where an observation lies compared with everyone else.
• Difference between "Absolute" scores and "relative" scores.
• Example: Reading comprehension score in the 3rd percentile.

Percentages

• When given percentages, be aware of information without base numbers.
• Receiving a letter claiming a company's profits were 46% higher than the year before.
• Consider if the firm earned 27 cents last year, which turned into 39 cents this year (only a 46% increase.)

The Standard Deviation

• Indicates how dispersed the data are from their mean/how spread apart are the observations.
• Example: Data was collected on the weights of 250 people on an airplane headed for Boston and the weights of 250 qualifiers for the Boston Marathon.
• However, the mean weight for both groups is 155 pounds, and the former will have a greater standard deviation.

Deceptive Descriptions

• The field of statistics is rooted in mathematics, and mathematics are exact.
• However, the use of statistics to describe complex phenomenom are not exact.
• Be sure to pay attention to the unit of analysis.
• There can be a lack of clarity over what one is defining, describing, or explaining.
• How does one define health as "manufacturing output" or "employment"?

Data Designs and Sampling Strategies

• Provenance: "the place of origin of something."
• Literary Digest accurately predicted the correct candidate to win the presidental race from 1916 - 1932.
• In 1936, about 40 million voters were expected.
• Literary Digest sent out 10 million mock ballots which it received 2.4 million back.
• The prediction was that Landon had 57% and Roosevelt 43% of the votes.
• The actual result was that Landon had 38% and Roosevelt 62% of the votes.
• A 1988 study showed that the 10 million voters who received mock ballots were not representative.
• They were found to be drawn from wealthier voters.
• The 2.4 million voters that responded were also found to be not representative.
• The most passionate were more likely to respond.
• There's a fundamental problem: Poor data provenance, as big data is equated to good data.
• Census: Information on all subjects.
• Subset or sample: Information on some subjects.
• Self-selected sample example: Subset is whoever chooses to answer.
• Convenience sample: Subset is whomever is convinient for the researcher.
• Judgement sample: Subset is whoever the researcher deliberately selects.
• Probability sample: Subset involves some probabilistic selection.
• Adminstrative dataset: A dataset collected as parts of administrative work ( e.g. social security names, restaurant safety ratings, etc.)

Simple Random Sample (SRS)

• A "sample drawn from population at random without replacment."
• If we have 6 people and want to sample them (A,B,C,D,E,F) the 15 samples of size 2 are AB, AD, AE, AF, BC, BD, BE, BF, CD, CE, CF, DE, DF, and EF.
• Then P(AB)=P(AC)=...P(EF)=1/15
• And P(A)=5/15=1/3

Cluster Sampling

• Divide the population into clusters and then SRS chooses clusters instead of individuals (example: everyone in a town)
• If we have 6 people and want to sample them (A,B,C,D,E,F) the first we've chosen are: A,B/ C,D/ E,F arbitrarily. Then SRS picks one cluster (A,B).
• Risk of bias if members of clusters are similar.
• Could be cheaper to use when polling everyone at an address.

Stratified Sampling

• It divides the population into clusters, and then creates the first SRS per strata.
• Gender Identification is an example (Female, Male, Other) First the strata is chosen: {A,B.C.D} and {E,F } arbitrarily.
• Then uses SRS to select one person from each strata.

First Borns and Harvard

• Harvard Philosopher Michael Sandel argues that birth order has a significant impact on work ethic.
• He found in class what fraction of students were first born, and found out to be consistently 75-80% born firstP(F|h)=75-80%
• US census data for mothers with children in college is:
• 1 Child: 21%, 2 Children: 43%, 3 Children: 23%, 4 Children: 13%
• And percent first born is: 100/(21+43 * 2 +23 * 3 + 13*4)=44 %
• Measuring PP(F|h)=80 % and comparint to 44% is insufficient.
• Harvard must consider how many children mother's Harvard tend to have.
• The fact that Harvard mothers tend to have very few children.
• The 2012 study walk throught the math regarding Sandels argument in great detail. Sandels Measures P(F|h)
• But what he cares about is a different quanity.
• Sandel wants to show that a first born conveys an advantage over NOT.
• He wants to show r is P(H|F) (the probability of getting into Harvard if first born) /P(H|FC) (the probability of getting into Harvard if not first born).
• P(H|F): Probability of getting into Harvard firstborn.
• P(H|FC): Probability of NOT getting into Harvard firstborn. The higher r is the better for the firstborn.

Applying BAyes RUle for Harvard first Borns P(F|H)

• Let's say r=3, that being first born makes you 3 times as likely to get into Harvard.
• If r > 1: it'd imply that being first born gives you "magic touch ( smart, assistance in demons...
• Find the CHance for first born to be at harvard.
• We want the change you are the harvard student (H) given your first born.
• Compare it to chane you are not first Born.
• The population of students are N first born where it is the fertitlity rate.
• IF the US Value for 1994 we get r=3.9
• But in the most elite schools in the US r=3.2

Notebooks in data science allow for narrative presentation with code, documentation, and markdown. Descriptive statistics present data meaningfully but can be misleading. Deceptive descriptions occur when statistics don't fully represent complex phenomena, such as using per capita income to assess the economic health of the middle class.