Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
data_analysis:ab_test [2021/06/05 20:28] prgram |
data_analysis:ab_test [2021/09/30 10:07] (current) prgram [A/B test] |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== ab test ====== | + | ====== A/B test ====== |
+ | |||
+ | AB 테스트 추천 도서 : [[https://www.amazon.com/Trustworthy-Online-Controlled-Experiments-Practical/dp/1108724264|Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing]] [[https://coupa.ng/b7Omnt|쿠팡]] | ||
+ | 통계적 지식 뿐 아니라 실제 적용에서 고민해야할 문제들에 대한 상세한 설명이 있다. | ||
[[data_analysis:convergence_testing]] | [[data_analysis:convergence_testing]] | ||
https://reflectivedata.com/comprehensive-guide-to-statistics-in-a-b-testing/ | https://reflectivedata.com/comprehensive-guide-to-statistics-in-a-b-testing/ | ||
- | https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf | ||
https://www.slideshare.net/cojette/ab-150118831 | https://www.slideshare.net/cojette/ab-150118831 | ||
https://onlinemix.tistory.com/entry/significant-result-from-ab-testing | https://onlinemix.tistory.com/entry/significant-result-from-ab-testing | ||
- | https://is.muni.cz/th/wt0tu/Humaj-thesis.pdf | ||
https://cxl.com/blog/ab-testing-guide/ | https://cxl.com/blog/ab-testing-guide/ | ||
https://hbr.org/2017/06/a-refresher-on-ab-testing | https://hbr.org/2017/06/a-refresher-on-ab-testing | ||
+ | Bayesian | ||
+ | https://is.muni.cz/th/wt0tu/Humaj-thesis.pdf | ||
+ | https://brunch.co.kr/@gimmesilver/15 | ||
+ | [[https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf|VWO_SmartStats_technical_whitepaper]] | ||
- | Sample size calculation for two independent Groups | + | |
- | https://www.researchgate.net/publication/287927836_Sample_Size_Calculation_for_Two_Independent_Groups_A_Useful_Rule_of_Thumb | + | [[https://www.researchgate.net/publication/287927836_Sample_Size_Calculation_for_Two_Independent_Groups_A_Useful_Rule_of_Thumb|Sample size calculation for two independent Groups]] |
https://www.evanmiller.org/ab-testing/ | https://www.evanmiller.org/ab-testing/ | ||
+ | |||
+ | MAB | ||
+ | http://sanghyukchun.github.io/96/ | ||
+ | https://kw94.tistory.com/49 | ||
+ | https://sumniya.tistory.com/9 | ||
+ | |||
+ | |||
+ | ===== 실행순서 ==== | ||
+ | * 가설 : 인터뷰, idea, 기존자료 분석 | ||
+ | * 실험설계 : control/treatment, 지표, bayesian/frequentist | ||
+ | * Random unit : user/session/page | ||
+ | * Target unit : 전체 / segment | ||
+ | * Size : sample, traffic 분배 | ||
+ | * How long : 학습효과 | ||
+ | * 실행 : 모니터링. 오류, 다른 feature들이 떨어지는지, 예상치 못하게 효율이 떨어지는지 (cf. 가드레일 metric) | ||
+ | * 분석 : traffic 분배 정합성, 샘플수, 다른 segments의 특성, 다른 지표, Bot은 없는지 | ||
+ | * 결정 : 통계적vs실질적 유의도, A/A test->검증 | ||
{{tag>data_analysis 실험 experiment ABtest}} | {{tag>data_analysis 실험 experiment ABtest}} |