unit of analysis - 看分母

1. sanity check for invariant metrics

问题：uber不同算法 - 选event还是user based？

metric sensitivity and robustness

- average is sensitive to outliers

- easy to move? (median is hard to change)

- standard deviation vary a lot?

Normal Distribution Steps - confidence interval

1. calculate p difference

2. calculate p pool

3. calculate pool standard deviation (if given empirical sd, use new sd/se)

4. calculate margin of error

5. calculate confidence interval

6. check statistical (default is 0)/practical (if given dmin) significance

问题：为什么smoking那道题normal distribution的顶是20%？不是10000人里有8000人抽烟吗

analytical variability - 通过ab测试的实验数据得出，容易underestimate （不复杂的normal distribution用这个）

如何计算：1. 算difference 2. p pool 3. SE_pool 4. margin of error 5. confidence interval

empirical variability - 通过以前的实验或者理论得出，适用于non-parametric metric

如何计算：1. 求difference 2. 用diff算stdv

问题：课最后的diff order怎么得出来的

segmentation - 看treatment group和control group有没有区别可以分channel看 （e.g.找工作不同的渠道）

event based: - 同一个人每次看会变 better when the change is not visible to avoid disruption of experience, short term

cookie (user) based - 看过这个feature的永远保持在一组: long-term, 人对于这个feature的影响

invariant metric

AARRR

acquisition

- customer acquisition cost CAC/CPA

- daily new users DNU

- daily one session users DOSU (一次开关算一个session)

activation

- dailyt/weekly/monthly active users 

- daily average online time DAOT/AT

retention and churn 保留和流失

- day 1/7/30 churn/retention rate

revenue

- active payment account APA

- average revenue per user ARPU (要大于cost per user才是好的)

- life time value LTV (只要一使用产品这个用户会贡献多少钱)

referral

Daily: 是否存在异常（节日，推广，版本更新，事故），渠道表现

weekly: 关注retention and churn

confidence intervals:

if want to use normal distribution, has to check N*p > 5 and N(1-p) > 5

margin of error=z*SE=z*sqrt(p(1-p)/n), z=1.96 if alpha=5%

confidence interval: [p-MOE, p+MOE]

Hypothesis test

alpha=P(rejected null | null is true) 错判

beta=P(fail to reject null | null is false) 漏网之鱼

power=1-beta

发现一个correlation之后可以反问：

1. how to define x and y

2. why is y variable important

3. scope of this obersation

什么情况不适合做ab test：

novelty effect: 视觉上的变化，用户可见的新功能性增加，比较难展开精准的ab test，如logo

不是所有产品都适合做ab test，比如卖车

stats考点： sample size, p value, std, minimum detectable effect, type 1/2 error, confidence interval, z test/score, t test/score, significant level, power rate

segment - 通常指把用户分成几类

click through rate (CTR) vs click through probability (distinct people)