What is Reddit's opinion of
Trustworthy Online Controlled Experiments?
From 3.5 billion Reddit comments
By popularity on Reddit, this product is:
16 reviews of this product found across Reddit:
If business related are acceptable, then I think Trustworthy Online Controlled Experiments is a great option
there’s the bible of experimental design: Trustworthy Online Controlled Experiments. Get it off amazon! Trustworthy Online Controlled Experiments (A Practical Guide to A/B Testing) https://smile.amazon.com/dp/1108724264/ref=cm_sw_r_cp_api_i_TGPWHWM02KC4NM37VED4?_encoding=UTF8&psc=1
> I do have analysis experience in research but not in an industry setting.
In that case, I would highly recommend familiarizing yourself with some basics around how industry and academia differ from each other. One book that's definitely worth going through is Trustworthy Online Controlled Experiments by Kohavi et al. Kohavi led experimentation at Bing, and discusses best practice around experimental design and analysis specifically in an online setting.
There are a ton of concerns in online A/B testing that are different from what most researchers encounter when designing experiments. For example:
If your experimental unit is "cookie", you don't actually know what your "users" are.
Online tests are typically performed in an iterative fashion where they test on like 1% of users, look for drastic changes, then ramp up to a larger percent to avoid implementation bugs and negative impact on users
In industry, people are often concerned with whether a percent change is statistically significant, which involves very different math compared to differences, where you have nice techniques like t-testing.
I've found in general that interviewers don't expect personal projects for A/B testing as much as they want to see if you can explain how to conduct an A/B test for a particular hypothesis.
If you know barely anything about experimental design, I'd recommend reading through Howard Seltman's Experimental Design and Analysis book. It's aimed at social scientists but a great resource in general.
If you know basic experimental design but don't know a lot about online A/B testing I recommend Trustworthy Online Controlled Experiments by Kohavi et al. It covers a lot of concepts specific to online A/B testing like indications that the platform itself is buggy, designing experiments where the same user doesn't see both the control and test experiences, and indications that your results aren't trustworthy.
Isn't a horrible place to start. It doesn't go very deep at all on the statistics involved, but does a great job of providing a solid framework for establishing a culture of experimentation.
Hi! This is actually discussed by the author of Trustworthy Online Controlled Experiments: https://www.amazon.com/Trustworthy-Online-Controlled-Experiments-Practical/dp/1108724264
He had a follow up that was really good, showed that the top product shops like Google, Airbnb, Microsoft, etc. have an average win rate of 10-15%. So we use that internally as a general guidance for our A/B testing program.
This might differ depending on how mature your product is. The less optimized the product, the higher this percentage could be. But on average the above makes sense as a benchmark.
Hope that helps!
I’ve been doing A/B testing for pretty much my whole career. Have done both in-house and at experimentation focused agencies. My focus has been mainly on the web but the foundation is still the same.
Ron Kohavi who has done testing at Airbnb, Microsoft, and Amazon has a great book caled Trustworthy Online Controlled Experiments
If you are a visual/auditory learner like myself, this video is extremely helpful for you to see the big picture as well as some specific examples of being a Growth PM, which is a term that some companies use these days.
Also as someone had mentioned, Georgi Georgiev is great and knows his stuff. He devised an agile calculator (cheap but not free) for testing that helps to make faster business decisions compared to other testing calculators out there. Not sure how much traffic your product gets but low traffic is always a hurdle for testing.
As others have also said, CXL is useful for quick learning and inspiration.
My tip is, don’t get caught up in the process too much. I made the mistake of spending too much time doing preparation for testing and not enough on actually getting tests out earlier in my career.
It’s important to have the knowledge but actually being effective in running tests is a whole another skillset. Testing velocity is arguably the most meaningful measure of an effective experimentation program. Your manager, his/her boss, and your peers will want to hear about the tests you ran, what you learned from it and what was gained from it. Of course many will want to see positive results on the KPIs but I’ve always emphasized the learning aspect more because the learnings can scale across the org. If you focus only on the key metrics being measured, you are also risking your performance to be reliant on your test wins, which will no doubt have its ups and downs.
Where to buy 2nd hand books?
Looking for this : https://www.amazon.com/Trustworthy-Online-Controlled-Experiments-Practical/dp/1108724264
People have already mentioned the books I was planning on recommending (FPP, ISL, Statistical Rethinking, Machine Learning a Probabilistic Perspective), so I'm going to take a slightly different approach. These are some of the books that I've found to be the most influential/useful in my personal career.
Perfect Pitch: The Art of Selling Ideas and Winning New Business I'm a bit hesitant to recommend a specific book here, but learning how to present and sell new ideas to stakeholders has been one of the hardest and most valuable skills I've learned. This is the only specific title I can remember.
Trustworthy Online Controlled Experiments is a great overview of A/B testing in a web environment.
Couple recommendations for you:
** A/B testing: **
- Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing - I looked for a free version of this, but I can't seem to find one. Definitely worth the purchase though!
** Intro to Statistics: **
Lots of good resources here, if you have no experience with coding I would recommend Dr Nic's Maths and Stats for a good excel based overview of several fundamental concepts
Khan Academy has some really good material.
** Google Analytics/Google Ads: **
- Google Analytics Academy for Google Analytics * Google Skillshop for other Google-y things
** Slack Communities: **
Measure.chat - probably the best community directly related to marketing analytics. Lots of really really smart people in here who are very helpful. I believe they do require you to be working in an applicable industry, but that specific rule might not be enforced all that much.
Locally Optimistic - slightly more data science/analytics specific than measure, but still has some really good content
Datatalks.club - mostly geared toward entry level data scientists/analysts, or those interested in transitioning into the industry.
** Subreddits: **
/r/dataanalysis - like datatalks, this one seems to be mostly geared toward more entry level data scientists/analysts, or those interested in transitioning into the industry. There are definitely some good posts here though!
Happy to help out!
The most straightforward and statistically rigorous way to determine causality is to run an experiment. So back to our SF example, you would randomly split your audience into a test and a control group. For your control group, you would leave everything the same while in your control group you would change the time to contact. After running your experiment for some predetermined time, you would look at the results and see if there was a statistically significant difference between the two. The difference is the causal effects of whatever you're measuring.
There are other ways using purely observational data (or historical data you've collected but not run any experiments on), but they get a little messy. You can have quasi-experimental methods like regression discontinuity, difference in difference that both rely specifically on linear regression, or you could start getting into some really nasty stuff with directed graphs.
At the end of the day, the idea remains pretty much the same for all these observational approaches. You're trying to predict what would have happened if you hadn't done something (determine the counterfactual prediction) and comparing it to what actually happened. The difference between the two numbers is the causal effects.
It's a really interesting but challenging problem space. Plenty of room for growth, and people who are good at it can be incredibly valuable.
If you're interested in learning more here are some resources for you. Some of these can get a little dense and have some scary math, but are good overviews of causal inference. Again it can be a challenging problem space and takes a little getting used to, but remember the core idea: All you're trying to do is predict what would have happened if you hadn't done the thing you did, then take the difference between the two. All the math and statistical tests are just trying to determine how well you predicted and if there is actually a difference between your prediction and the thing that happened.
Difference in Difference Lecture Notes
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing - I couldn't find a free version, but it is pretty cheap and a fantastic read.
Presentation notes for Trustworthy Online Controlled Experiments
There is a glossary of terms, see: https://www.analytics-toolkit.com/glossary/.
As for a complete guide/dict, see https://www.amazon.com/Trustworthy-Online-Controlled-Experiments-Practical/dp/1108724264
This book has a pretty good framework for different levels of maturity of the A/B testing process. Highly recommended.
As an compromise you can try to take this series of courses (https://www.edx.org/micromasters/mitx-statistics-and-data-science) I'm currently taking the mathematical statistics course and enjoying it.
I see a lot of Masters/PhDs from other disciplines do well as data scientists at my company - they mostly self learn and the area for these PhDs are usually in business/product strategy.
I would strongly recommend getting good at SQL - this is the bread a butter (you wrangle and aggregate data in SQL before pulling into R/python) and experimental design (A/B testing).
https://mode.com/sql-tutorial/introduction-to-sql/
Popular alternatives to Trustworthy Online Controll...
Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are
