发现一个correlation之后可以反问：

1. how to define x and y

2. why is y variable important

3. scope of this obersation

什么情况不适合做ab test：

novelty effect: 视觉上的变化，用户可见的新功能性增加，比较难展开精准的ab test，如logo

不是所有产品都适合做ab test，比如卖车

stats考点： sample size, p value, std, minimum detectable effect, type 1/2 error, confidence interval, z test/score, t test/score, significant level, power rate

segment - 通常指把用户分成几类

click through rate (CTR) vs click through probability (distinct people)