{"id":511,"date":"2022-10-28T12:11:37","date_gmt":"2022-10-28T12:11:37","guid":{"rendered":"https:\/\/analysisparalysis.org\/?p=511"},"modified":"2022-10-29T08:22:18","modified_gmt":"2022-10-29T08:22:18","slug":"temporal-limitations-of-ab-testing","status":"publish","type":"post","link":"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/","title":{"rendered":"The Temporal Limitations of AB Testing"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"511\" class=\"elementor elementor-511\">\n\t\t\t\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-67d7b5ae elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"67d7b5ae\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-7f4160ef\" data-id=\"7f4160ef\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-3482e98e elementor-widget elementor-widget-text-editor\" data-id=\"3482e98e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<style>\/*! elementor - v3.18.0 - 20-12-2023 *\/\n.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block}<\/style>\t\t\t\t<p><!-- wp:paragraph {\"align\":\"right\"} --><\/p>\n<p dir=\"ltr\" style=\"line-height: 1.3800000000000001; margin-top: 0pt; margin-bottom: 0pt;\"><span style=\"font-size: 11pt; font-family: Arial; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">With every post I write, I wonder what direction it will take this blog, and what value it provides to analysts. And although I often write about career, I feel like it is lacking more professional content. <\/span><\/p>\n<p dir=\"ltr\" style=\"line-height: 1.3800000000000001; margin-top: 0pt; margin-bottom: 0pt;\"><span style=\"font-size: 11pt; font-family: Arial; color: #000000; background-color: transparent; font-weight: 400; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">Now, there are reasons for that, obviously \u2014 It started as a Facebook page, where the platform doesn\u2019t allow putting images into the text, so every point had to be explained verbally, which is sometimes complex, but anyway \u2014 I wrote a three-post series on a professional topic, so let\u2019s hope it works out!<\/span><span style=\"background-color: transparent; color: #000000; font-family: Arial; font-size: 11pt; white-space: pre-wrap;\">\u00a0<\/span><\/p>\n<p>\u00a0<\/p>\n<p><!-- \/wp:paragraph --><\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-8151098 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"8151098\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-d123722\" data-id=\"d123722\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-fbf2c66 elementor-align-center elementor-widget elementor-widget-button\" data-id=\"fbf2c66\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"#\" target=\"_blank\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">The Greediness of AB      Tests       <\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-3ecdf5a\" data-id=\"3ecdf5a\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-a581772 elementor-align-center elementor-widget elementor-widget-button\" data-id=\"a581772\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-md\" href=\"#\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t<span class=\"elementor-button-text\">The Intransitiveness of AB Tests<\/span>\n\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-90bee24 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"90bee24\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-40f7a22\" data-id=\"40f7a22\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-46cf942 elementor-widget elementor-widget-text-editor\" data-id=\"46cf942\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<p>Sometime recently (just this week, as of writing the draft for this post) I was in a Monday conference, organized by Eldar Muzikansky, which had two interesting lectures, by Allon Korem and Michael Elbaz.<\/p>\n<p>\u00a0<\/p>\n<p>Both lectures were enriching, touching on plenty of\u00a0 relevant and interesting points, but (as both lecturers would agree) didn\u2019t cover even 0.1% of the interesting problems in the field \u2014 giving me some inspiration. So let\u2019s talk for a bit not about how to perform A\/B testing, but about their limitations.<\/p>\n<p>\u00a0<\/p>\n<p>This post would cover the issue of temporal limitations for A\/B testing<\/p>\n<p>\u00a0<\/p>\n<p>I would to state, right off the bat, that all of the examples I\u2019m about to give here are silly. Obviously because I\u2019m trying to make a point, but I hope that we can learn from that point, identify a pattern, and from that pattern start a discussion with ourselves or team members running the test.<\/p>\n<p>\u00a0<\/p>\n<p>For context, let us imagine we have an operation team whose only role is feeding text into web pages, with the hope that during a google search (what we call in the biz \u201corganic search\u201d) it would attract to the page the most relevant audience for producing a purchase (or any other kind of conversion, doesn\u2019t really matter).<\/p>\n<p>\u00a0<\/p>\n<p>So this team examines their site on August, and comes up with the following hypothesis:<\/p>\n<p>\u201cIf towards the end of September we start placing Christmas related terms over the site, these pages would show up higher on searches for expressions related to the product and Christmas, giving us more relevant entries.\u201d<\/p>\n<p>\u00a0<\/p>\n<p>They define the results perfectly and when they get them decide to run a dependent sample t-test for a before and after diagnosis, and after running all of the analytic work we find that the after group is markedly higher and it\u2019s statistically significant and hooray and good work everyone \u2014 everybody is happy, they present it to management, get an applause and here\u2019s the question now: what is the conclusion of the experiment?<\/p>\n<p>\u00a0<\/p>\n<p>The conclusion is, of course, the more Christmas related terms we put on our pages, the higher they\u2019ll rank.<\/p>\n<p>\u00a0<\/p>\n<p>The team members are happy, they write a case study and even present it during a meetup with pizza and beer and networking and all that.<\/p>\n<p>\u00a0<\/p>\n<p>During the meetup, a team from Another Organizations hears about the experiment and decides to implement the changes as written \u2014 which results in Another Organization\u2019s site getting a Christmasy makeover \u2014 a sprinkle of Christmas terms, red and green color scheme, ho-ho-ho and all that \u2014 in May.<\/p>\n<p>\u00a0<\/p>\n<p>Now, we\u2019re all clear that the result of this experiment (at least on the immediate surface level) won\u2019t be beneficial to them, right?<\/p>\n<p>Imagine yourself going into some commerce site full of bats and jack-o-lanterns, but what you\u2019re looking for is chocolate eggs (full disclosure \u2014 I\u2019m always looking for chocolate, eggs or otherwise).<\/p>\n<p>The problem is that the conclusions from the experiment weren\u2019t limited with regards to the time axis, and although I gave a classic example, there\u2019s no reason to assume that the time axis doesn\u2019t influence all other tests we perform.<\/p>\n<p>\u00a0<\/p>\n<p>We can expand on this point and say we all know the issue \u2014 after all, one of the explicit reasons we use a two-three weeks duration is to express weekly seasonality (because behavior on weekdays and weekends can be different)<\/p>\n<p>\u00a0<\/p>\n<p>So if we can agree with the idea that conclusion from A\/B testing are limited in terms of time, we can\u2019t continue to argue that \u201cthe change group performs better than the control\u201d without considering significant changes in terms of times and population characteristics.<\/p>\n<p>\u00a0<\/p>\n<p>What am I suggesting? That when we present the conclusion of an A\/B test, we have to present all information that could have influenced and can qualify the conclusions.<\/p>\n<p>\u00a0<\/p>\n<p>In terms of demographics:<\/p>\n<p>If we created a test and, completely by coincidence, only teenagers took part in it, we can\u2019t apply the conclusions to other age-groups.<\/p>\n<p>If my test, by complete coincidence, had a majority of male participants, applying the conclusion to the entire population is problematic to the point of dangerous (I\u2019m looking at you, clinical trials with female under-representation).<\/p>\n<p>\u00a0<\/p>\n<p>In terms of time axis:<\/p>\n<p>Merry Christmas and ho-ho-ho notwithstanding, the purchase of certain items can be influenced by other significant events, like say a war in Europe, even without\u00a0 a direct connection (but also tread lightly with the idea of \u201cthe butterfly effect\u201d and how everything can influence everything \u2014 keep your limitations reasonable).<\/p>\n<p>\u00a0<\/p>\n<p>In terms of our system:<\/p>\n<p>If we examined a certain element in the page, but our back-end system changed since then &#8211; the results of that test might not be accurate or reliable anymore.<\/p>\n<div><span style=\"font-size: 11pt; font-family: Arial; color: #000000; background-color: transparent; font-variant-numeric: normal; font-variant-east-asian: normal; vertical-align: baseline; white-space: pre-wrap;\">\u00a0<\/span><\/div>\n<div>In conclusion, I&#8217;m not saying that I&#8217;m an AB testing expert, because I&#8217;m not, but I do want you to pay close attention to how you use your tools.<\/div>\n<div>\u00a0<\/div>\n<div>\u00a0<\/div>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>With every post I write, I wonder what direction it will take this blog, and what value it provides to analysts. And although I often write about career, I feel like it is lacking more professional content. Now, there are reasons for that, obviously \u2014 It started as a Facebook page, where the platform doesn\u2019t [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":626,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23],"tags":[],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>The Temporal Limitations of AB Testing - Analysis Paralysis<\/title>\n<meta name=\"description\" content=\"Let&#039;s discuss how our conclusions from AB testing are too lenient - especially if we ignore time effects on our results\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The Temporal Limitations of AB Testing - Analysis Paralysis\" \/>\n<meta property=\"og:description\" content=\"Let&#039;s discuss how our conclusions from AB testing are too lenient - especially if we ignore time effects on our results\" \/>\n<meta property=\"og:url\" content=\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"Analysis Paralysis\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-28T12:11:37+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-10-29T08:22:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/analysisparalysis.org\/wp-content\/uploads\/2022\/10\/AB-Testing-santas.jpeg\" \/>\n\t<meta property=\"og:image:width\" content=\"340\" \/>\n\t<meta property=\"og:image:height\" content=\"225\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"talnmizrachi@gmail.com\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"talnmizrachi@gmail.com\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/\"},\"author\":{\"name\":\"talnmizrachi@gmail.com\",\"@id\":\"https:\/\/analysisparalysis.org\/#\/schema\/person\/44b6068673947340d0f78950bb3070a8\"},\"headline\":\"The Temporal Limitations of AB Testing\",\"datePublished\":\"2022-10-28T12:11:37+00:00\",\"dateModified\":\"2022-10-29T08:22:18+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/\"},\"wordCount\":939,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/analysisparalysis.org\/#organization\"},\"articleSection\":[\"Professional\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/\",\"url\":\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/\",\"name\":\"The Temporal Limitations of AB Testing - Analysis Paralysis\",\"isPartOf\":{\"@id\":\"https:\/\/analysisparalysis.org\/#website\"},\"datePublished\":\"2022-10-28T12:11:37+00:00\",\"dateModified\":\"2022-10-29T08:22:18+00:00\",\"description\":\"Let's discuss how our conclusions from AB testing are too lenient - especially if we ignore time effects on our results\",\"breadcrumb\":{\"@id\":\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/analysisparalysis.org\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The Temporal Limitations of AB Testing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/analysisparalysis.org\/#website\",\"url\":\"https:\/\/analysisparalysis.org\/\",\"name\":\"Analysis Paralysis\",\"description\":\"Carpe Datum\",\"publisher\":{\"@id\":\"https:\/\/analysisparalysis.org\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/analysisparalysis.org\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/analysisparalysis.org\/#organization\",\"name\":\"Analysis Paralysis\",\"url\":\"https:\/\/analysisparalysis.org\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/analysisparalysis.org\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/analysisparalysis.org\/wp-content\/uploads\/2022\/08\/cropped-cropped-Analysis-Paralysis-logo.png\",\"contentUrl\":\"https:\/\/analysisparalysis.org\/wp-content\/uploads\/2022\/08\/cropped-cropped-Analysis-Paralysis-logo.png\",\"width\":549,\"height\":116,\"caption\":\"Analysis Paralysis\"},\"image\":{\"@id\":\"https:\/\/analysisparalysis.org\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/analysisparalysis.org\/#\/schema\/person\/44b6068673947340d0f78950bb3070a8\",\"name\":\"talnmizrachi@gmail.com\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/analysisparalysis.org\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/1ebeb6e3094dd9013d20bf5d466c26ef?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/1ebeb6e3094dd9013d20bf5d466c26ef?s=96&d=mm&r=g\",\"caption\":\"talnmizrachi@gmail.com\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The Temporal Limitations of AB Testing - Analysis Paralysis","description":"Let's discuss how our conclusions from AB testing are too lenient - especially if we ignore time effects on our results","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/","og_locale":"en_US","og_type":"article","og_title":"The Temporal Limitations of AB Testing - Analysis Paralysis","og_description":"Let's discuss how our conclusions from AB testing are too lenient - especially if we ignore time effects on our results","og_url":"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/","og_site_name":"Analysis Paralysis","article_published_time":"2022-10-28T12:11:37+00:00","article_modified_time":"2022-10-29T08:22:18+00:00","og_image":[{"width":340,"height":225,"url":"https:\/\/analysisparalysis.org\/wp-content\/uploads\/2022\/10\/AB-Testing-santas.jpeg","type":"image\/jpeg"}],"author":"talnmizrachi@gmail.com","twitter_card":"summary_large_image","twitter_misc":{"Written by":"talnmizrachi@gmail.com","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/#article","isPartOf":{"@id":"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/"},"author":{"name":"talnmizrachi@gmail.com","@id":"https:\/\/analysisparalysis.org\/#\/schema\/person\/44b6068673947340d0f78950bb3070a8"},"headline":"The Temporal Limitations of AB Testing","datePublished":"2022-10-28T12:11:37+00:00","dateModified":"2022-10-29T08:22:18+00:00","mainEntityOfPage":{"@id":"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/"},"wordCount":939,"commentCount":0,"publisher":{"@id":"https:\/\/analysisparalysis.org\/#organization"},"articleSection":["Professional"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/","url":"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/","name":"The Temporal Limitations of AB Testing - Analysis Paralysis","isPartOf":{"@id":"https:\/\/analysisparalysis.org\/#website"},"datePublished":"2022-10-28T12:11:37+00:00","dateModified":"2022-10-29T08:22:18+00:00","description":"Let's discuss how our conclusions from AB testing are too lenient - especially if we ignore time effects on our results","breadcrumb":{"@id":"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/analysisparalysis.org\/temporal-limitations-of-ab-testing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/analysisparalysis.org\/"},{"@type":"ListItem","position":2,"name":"The Temporal Limitations of AB Testing"}]},{"@type":"WebSite","@id":"https:\/\/analysisparalysis.org\/#website","url":"https:\/\/analysisparalysis.org\/","name":"Analysis Paralysis","description":"Carpe Datum","publisher":{"@id":"https:\/\/analysisparalysis.org\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/analysisparalysis.org\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/analysisparalysis.org\/#organization","name":"Analysis Paralysis","url":"https:\/\/analysisparalysis.org\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/analysisparalysis.org\/#\/schema\/logo\/image\/","url":"https:\/\/analysisparalysis.org\/wp-content\/uploads\/2022\/08\/cropped-cropped-Analysis-Paralysis-logo.png","contentUrl":"https:\/\/analysisparalysis.org\/wp-content\/uploads\/2022\/08\/cropped-cropped-Analysis-Paralysis-logo.png","width":549,"height":116,"caption":"Analysis Paralysis"},"image":{"@id":"https:\/\/analysisparalysis.org\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/analysisparalysis.org\/#\/schema\/person\/44b6068673947340d0f78950bb3070a8","name":"talnmizrachi@gmail.com","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/analysisparalysis.org\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/1ebeb6e3094dd9013d20bf5d466c26ef?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ebeb6e3094dd9013d20bf5d466c26ef?s=96&d=mm&r=g","caption":"talnmizrachi@gmail.com"}}]}},"lang":"en","translations":{"en":511,"he":486},"pll_sync_post":[],"_links":{"self":[{"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/posts\/511"}],"collection":[{"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/comments?post=511"}],"version-history":[{"count":37,"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/posts\/511\/revisions"}],"predecessor-version":[{"id":653,"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/posts\/511\/revisions\/653"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/media\/626"}],"wp:attachment":[{"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/media?parent=511"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/categories?post=511"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/analysisparalysis.org\/wp-json\/wp\/v2\/tags?post=511"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}