Beginners Guide to Hypothesis Testing

What is hypothesis testing?

Assume that we have a coffee shop. Since we always look for ways to improve the business, we decided to offer free wifi for our customers. We believe customers spent more time in the shop after the introduction of this new change. The data collected also shows an increase in the meantime spent by the customers. But wait, how can we make sure that it is not due to pure random chance?

This is where hypothesis tests are used. It helps us to make sure that we are giving in to some random effects. We make a null and alternate hypothesis statement to test the effect. After collecting the data of time spent by customers, before and after the change, we can use one of the statistical methods available to verify the effect. The most appropriate one for this problem is Z-test. P-value is very useful and helps in determining the chance of having the result due to pure random chance. The lower the p-value, the lower the chance of randomness. Accepting or rejecting the change depends on the significance level we set. Here we use test statistic instead of P-value. Let's solve our cafe problem by using hypothesis testing.

Significance Level:

The significance level is the probability of rejecting the null hypothesis when it is true. In other words, the level of risk we are willing to take. The most common significance level is 0.05, but 0.01 is prevalent in the pharma industry.

Formulate the hypothesis:

Ho = The mean time spent by customers unchanged (Null Hypothesis)

Ha = The mean time spent by customers are not equal (Alternate Hypothesis)

Z-Score:

After formulating the hypothesis and get our hands on the data, we can start calculating the Z-score. The formulae for Z-Score is 

z = (x-mean)/(standard deviation/sqrt(n))

x = The mean that we found from our data after introducing the change.

mean = Calculated mean before introducing the change.

Standard deviation = Standard deviation of our data collected after the change.

n = No of the samples selected.

After finding the Z-test statistic, we can use the Z-distribution curve to find the corresponding Z-value for the significance level we have chosen. For a significance level of 0.05, the Z-value is 1.96 for a two-tail test. If the Z-test statistic is less than 1.96 we failed to reject the null hypothesis, which means introducing wifi has no significant effect on the increased customer time spending. If the Z-test statistic is greater than 1.96, then we reject the null hypothesis and the results are statistically significant.

Assumptions to Check:

There are four main assumptions we make in this test.

  1. The data are continuous.
  2. The data follow a normal probability distribution.
  3. The population standard deviation is known.
  4. The samples are randomly selected.

If our model violate one of these assumptions, we cannot use the Z-test for this testing.

If you would like to know more about hypothesis testing, I suggest taking a look at 'Design and Analysis of Experiments' by Douglas Montgomery.

Comments

Post a Comment

Popular posts from this blog

Pydantic and Large Language Models: A High-Level Overview

Image
Large Language Models (LLMs) such as GPT-5.2 or Gemini 3 Pro deliver powerful natural language understanding and generation capabilities. However, one of the key challenges with LLMs in practical applications is managing their outputs, which are often unstructured, inconsistent, and unpredictable. This makes it difficult to integrate them reliably into real-world systems that require precise, validated data. Pydantic, a Python library built around the concept of defining data models with type annotations, provides a compelling solution by offering runtime data validation and structured data parsing. When used alongside LLMs, Pydantic models help turn the free-form language output into well-defined, predictable data structures. Why Pydantic Matters for LLMs LLM outputs can vary widely-even for the same prompt-with differences in formatting, missing fields, or unexpected values. This variability poses risks of processing errors, faulty logic, or incorrect data interpretation in applicati...
Read more

Welcome

Hey everyone, and welcome to my data science and tech space! I'm Pradeep Saminathan, a data scientist with a serious love for all things data and tech. I've been thinking about how to best share my passion and knowledge, and that's when I decided to really get this blog going. It's been around for a while, but now I'm ready to make it something special. What's in store for you? I'll be diving into the concepts and ideas that I find most interesting and important in data science and tech. This isn't your typical motivational blog – it's about practical knowledge and exploration. If you're as passionate about data science as I am, I'd love for you to join me! I'm always learning, and I'd love to have you along for the ride. My LinkedIn: https://www.linkedin.com/in/pradeepos/
Read more

Top 5 LLMs, according to LLMs

  Given the rapid evolution of the LLM landscape and the continuous innovation by key players to push AI boundaries, I was curious to see how different LLMs would respond to inquiries about top LLMs. Specifically, I wanted to explore if there are any potential biases. The following are the outputs from various LLMs I tested (from free versions). OpenAI/ ChatGPT's Perspective: OpenAI Innovations : Developed ChatGPT, with iterations including GPT-4, GPT-4.5, and future versions. Collaborations : Partnered with Microsoft to integrate OpenAI models into Azure AI and Copilot. Google DeepMind Developments : Introduced Gemini (formerly Bard), a robust multimodal AI model. Research Focus : Engages in extensive research on scalable and ethical AI. Meta (Facebook AI) Contributions : Created the LLaMA series (LLaMA 2, LLaMA 3), offering open-weight LLMs. Philosophy : Emphasizes open-source AI development. Anthropic ...
Read more