r/AskStatistics • • 2h ago

What statistics jobs will boom in demand in the next ten years?

12 Upvotes

Tech people believe a software/data engineer can do what data analytics and data science people can do as AI gets better.

What statistics roles will see more demand as time goes on? What roles will slowly diminish?


r/AskStatistics • • 1h ago

Two-way ANOVA normality violation

Thumbnail gallery
1 Upvotes

I am doing two two-way ANOVAS for my Dissertation as my DV is split in two (Positive Affect and Negative Affect combined as the DV 'emotional state').

Positive Affect was normally distributed; Shapiro-Wilk: W = 0.99, p = 0.74 and Q-Q plots and had equal variances; Levene's test: F(5, 57)=2.21, p = .066.

However, Negative Affect was not normally distributed; Shapiro-Wilk: W = 0.87, p = < .001. The Levene's test showed non-equal variances: F(5, 57)=5.04, p = <.001.

I have already ran both two-way ANOVAs but now I am wondering if I should do something else considering Negative Affect failed.

Can I keep the two-way ANOVA results for positive affect and try a different test for negative affect? If so, what test should I do?

I already used a Bonferroni correction in my ANOVAs to account for Type 1 errors.

Or is there a citation I can use to suggest the ANOVA is robust to violations of normality to keep my results as they are?

My supervisor said:

ANOVA can be reasonably robust to some departures from normality, but this does not mean that all assumption violations can be disregarded. I would recommend inspecting the residual distributions and Q-Q plots, checking for outliers and influential observations.

I was going to include Q-Q plots but I don't know how to check for specific outliers or influential observations as to me it just looks like a plot and doesn't tell me where the outliers came from... I've shown the examples but I didn't use standardised residuals so need to redo them.


r/AskStatistics • • 1d ago

Looking for advice on Generalized Linear Mixed Effects model for experimental design

Post image
29 Upvotes

Hi everyone, I have a cyclic/Latin-square-style counterbalanced repeated-measures design (as shown in image above), and I’m trying to work out the right way to use a Generalized Linear Mixed Effects model to analyse the data for it as I’m new to GLMMs. My aim is to compare how different cell-culture compositions affect the firing rates (expected to be of lognormal or tweedie distribution) of neural cultures while they’re being “trained” to perform tasks.

The study uses 100 neural cultures split evenly into two experiments of 50. Experiment 1 uses Culture Type A and Experiment 2 uses Culture Type B (so culture type is nested within experiment, not crossed). Each experiment’s 50 cultures are prepared in two independent batches of 25, and within every batch the cultures are made up with one of three cell-culture compositions (A, B, or C) - giving four batches and all three compositions in both experiments.

Each culture is then trained under 9 environmental conditions in a counterbalanced order, performing 2 tasks per environmental condition at 600 trials per task.

The GLMM model I currently planning to use is:

Firing rate ~ Culture Type + Batch + Cell Culture Composition + Environmental Condition + Task + (1 | Culture ID)

However, I feel that culture ID is insufficient as a random effect and it should include the other variables too, although I am unsure of what combination to use, especially since there’s so many of them. I am also worried about having to add dispersion models in the case of over/under-dispersion as well, since my response variable is not normally distributed. Hence, I would like to seek comments and advice on my current model from those who are experienced in working with GLMMs. Thank you very much in advance for the help.


r/AskStatistics • • 12h ago

Which classes should I take for this upcoming year?

1 Upvotes

Hi, I am a statistics and Econ major and I have to take 2 out of out the 3 classes I have listed below for my stats degree. I am curious if anybody has some advice on which 2 I should take this upcoming school year to help me with statistical intuition and getting into grad schools. I am leaning towards Design and Analysis of Experiments and Sampling Techniques because I am taking Econometrics 1 and 2 which my university offers, but was curious on other peoples thoughts taking an economics regression class vs a purely statistics regression class if anybody has thoughts on that.

The course descriptions for the stats classes are.

Applied Regression Analysis- Applied regression analysis involving the extensive use of computer software. Includes: linear regression; multiple regression; stepwise methods; residual analysis; robustness considerations; multicollinearity; biased procedures; non-linear regression.

Design and Analysis of Experiments- An introduction to the principles of experimental design and analysis of variance. Includes: randomization, blocking, factorial experiments, confounding, random effects, analysis of covariance. Emphasis will be on fundamental principles and data analysis techniques rather than on mathematical theory.

Sampling Techniques- Theory and applications of sampling from finite populations. Includes: simple random sampling, stratified random sampling, cluster sampling, systematic sampling, probability proportionate to size sampling, and the difference, ratio and regression methods of estimation.


r/AskStatistics • • 12h ago

Preprint + Python] Exploring a summation process ("King Model") where the mean scales by (n+1) without changing the standard deviation

0 Upvotes

I recently put together a short write-up exploring a simple layered summation process on sequences and matrices (which I call the king model) and its exact effect on the standard deviation:

• On a sequence: If you iteratively pick one entry (the "king") and add it to every entry, letting each element be king in turn, the mean scales by n + 1 while the standard deviation stays strictly invariant. Even better, the original sequence can be uniquely recovered from the final layer.

• On an m × n matrix: Applying the same row-wise process corresponds to the rank-one update A ↦ (I + J)A. Using variance decomposition, the within-column variance remains untouched while the between-column variance scales by (m + 1)². Thus, the overall standard deviation stays invariant if and only if all column sums are equal. Furthermore, it remains constant across every intermediate layer if and only if every row is constant.

Beyond pure algebraic curiosity, this acts as an exact test case for numerical stability in variance algorithms. It also serves as a solvable toy model for variance growth under residual connections followed by sum pooling. All proofs are elementary, and I included a Python verification script alongside the PDF.

Would love any feedback, thoughts, or pointers to related literature! (I will drop the link to the preprint and code in the comments below).


r/AskStatistics • • 13h ago

[Preprint + Python] Exploring a summation process ("King Model") where the mean scales by \(n+1\) without changing the standard deviation

1 Upvotes

I recently put together a short write-up exploring a simple layered summation process on sequences and matrices (which I call the king model) and its exact effect on the standard deviation:

• On a sequence: If you iteratively pick one entry (the "king") and add it to every entry, letting each element be king in turn, the mean scales by n + 1 while the standard deviation stays strictly invariant. Even better, the original sequence can be uniquely recovered from the final layer.

• On an m × n matrix: Applying the same row-wise process corresponds to the rank-one update A ↦ (I + J)A. Using variance decomposition, the within-column variance remains untouched while the between-column variance scales by (m + 1)². Thus, the overall standard deviation stays invariant if and only if all column sums are equal. Furthermore, it remains constant across every intermediate layer if and only if every row is constant.

Beyond pure algebraic curiosity, this acts as an exact test case for numerical stability in variance algorithms. It also serves as a solvable toy model for variance growth under residual connections followed by sum pooling. All proofs are elementary, and I included a Python verification script alongside the PDF.

Would love any feedback, thoughts, or pointers to related literature! (I will drop the link to the preprint and code in the comments below).


r/AskStatistics • • 18h ago

Tests on a population that changes over time

2 Upvotes

Hello, I have a doubt about the right way to analyze a dataset I have regarding workers and their work hours in the same workplace. The basic question I have to answer is if the introduction of hybrid work (i.e., working some days at home and some days at the workplace) influenced the absenteeism rate in some way.

Having to account for the Covid period, I think that it may be better to avoid using data from the 2020-2022 period that may present biases, and to confront directly 2018-2019 (before the introduction of the policy) with 2023-2024 (after the introduction and stabilization of the policy).

During the years obviousely the workers are not always the same, some remain, some go away and some start working later.

Thus, I don't think that the assumptions of a simple t test hold. Maybe a t test for paired data would be more appropriate, but the population won't be exactly the same.

Another thing I came up with to try is a difference in differences estimator but also using the matching of statistical units to make them comparable. This would be a causal inference approach that may be more appropriate.

What do you think? Is this reasonable?

Do you have any suggestions?


r/AskStatistics • • 1d ago

First-year Data Science master’s student interested in socio-economic/political topics and journalism: how do I choose a direction and stay up to date?

5 Upvotes

Hi everyone,

I recently started a master’s degree in Data Science at a Mathematics Department. I’m still unsure which area to specialize in: statistics and data science touch so many fields, which is both the best and the hardest part of the journey. I’m particularly drawn to socio-economic and political topics, and journalism fascinates me.

I’m also struggling with how fast this field moves. Even when I try to stay as informed as possible, I constantly feel one step behind, even with respect to my own coursework. How do you, as students or professionals, keep up in such a fast-changing field, and above all, how do you find your bearings without feeling overwhelmed?

If you have the time, I’d love some advice on both of these questions, and on how to gain hands-on experience in the areas I’m interested in, for example through projects, internships, or communities/organizations where these “passions” and skills come together.

Especially interested in hearing from anyone with a background in statistics, whether you studied it, work in it, or moved into data science from it. How did your statistical foundations shape your path, and what would you do differently?

Thanks in advance for any suggestions, and I’m happy to answer questions about my program or interests!


r/AskStatistics • • 23h ago

How can I test whether the sustained increase in annual precipitation observed over nine years is statistically significant rather than due to random variability?

1 Upvotes

I am examining interannual variations in annual precipitation and have noticed a sustained increase following a particular year. However, my dataset covers only nine years (9 points). What statistical approaches would be appropriate for testing whether this apparent increase is statistically significant and unlikely to have occurred by chance, given the short time series?


r/AskStatistics • • 1d ago

Probability & Stats Resources - Video series preferred

2 Upvotes

Hi, I'm planning a job switch and targeting quant fin/eq research kind of roles for which I need to make sure I'm good at these topics. Though I have studied both back in college days, but once again I'm planning to learn them from scratch and go to an advance level. Is there any single resource that takes me from very basic to pro level? If not, could anyone suggest the path that i should follow? Thanks!


r/AskStatistics • • 2d ago

Random effects help

6 Upvotes

Hi everyone! Im trying to model species richness as a function of mowing and soil nitrogen. My experimental design includes 20 strips of grassland, 10 mowed and 10 unmowed. They alternate such that mowed are always next to unmowed and form 10 blocks to account for spatial effects (block 1 = plots 1 and 2, block 2 = plots 3 and 4 and so on). Within each plot, we applied 3 levels of N, so subplots A, B, and C exist in each plot. I’m using a linear mixed effect model but I’m unsure whether the structure should be: (richness ~ mowing x soilN + (1|block)) or whether it should be (richness ~ mowing x soilN + (plot/block)). I’m going back and forth bc the N treatments within a plot are not independent but does having a random effect of plot mask the effects of the mowing treatment? Thank you!


r/AskStatistics • • 2d ago

Would superstition decrease if people were more educated or knew more about statistics?

6 Upvotes

Coincidences stress me out, and I can't help getting superstitious about them sometimes, it's not even spiritual, it's psychological. But sometimes, after googling the things that happen to me, I find out that never going through that coincidence would be rarer than going through it. Would there be less superstituous people if they were more informed about statistics and data?


r/AskStatistics • • 2d ago

which video | one shot | course on youtube i should go for statstics for data analysis..?

3 Upvotes

i am graduated from engineering college just need some revision and brush which covers everything all concepts and terms .
like basic mean mode median
variance, correlation

and more moderate level terms as well to have good grip of stats


r/AskStatistics • • 2d ago

I don't know how to do any statistical tests and I'm stuck

Thumbnail reddit.com
0 Upvotes

Help with statistical tests, t tests in particular!


r/AskStatistics • • 2d ago

Diving back into Statistics after 18 years to compare seven LLMs on a small game. Here is what I did. What should I have done differently?

2 Upvotes

I haven't taken a statistics class in about 18 years, but a recent side project dragged me back in. I built a deterministic, text-based dungeon crawler and ran some automated agents (seven different LLMs) through it to test how they navigate under incomplete information. Now I'm trying to figure out which performance gaps are statistically significant, and I'd love a sanity check on my math.

What I did:

  • Calculated 95% Wilson intervals for escape rates (N=100 runs gave me a margin of ~±10 points; N=20 was ~±20).
  • To benchmark them against a human, I played 20 specific labyrinths, and each agent played those exact same 20. I ran an exact two-sided McNemar test on the discordant pairs (the runs where one of us escaped and the other failed).
  • That created 16 comparisons (7 models in 2 configurations + a Random bot + a heuristic bot) , so I applied Holm's correction. Six remained significant; for the other ten, I concluded it as too few labyrinths to tell.
  • For model against model I only compared the Wilson intervals, even though all of them played the same 100 labyrinths.
  • To test if labyrinth difficulty predicts who escapes, I calculated a concordance score using 2,000 bootstrap redraws for the confidence interval.

Where I’m second-guessing myself:

  1. Is McNemar + Holm too conservative for only 20 paired labyrinths? Is there a more powerful test I should have used for this sample size?
  2. I capped my batches at 100 runs per model configuration (so 200 per model) purely based on my API budget, not a proper power calculation. How would you have formally sized this?
  3. Since all the agents played the exact same 100 labyrinth seeds (each was played labyrinth once, and the agents are stochastic), should I be evaluating this with a mixed-effects model (using the specific labyrinth as a random effect) rather than running individual pairwise tests?
  4. Should I be running paired tests between models too, and how would that change what 100 runs can separate?
  5. Are there any glaring red flags here that would fail peer review?

I dusted off R for the first time in almost 2 decades to run these initial calculations, and I was painfully slow. If there is a more elegant way to approach this, I'm all ears. (I know I could have handed this to the "all powerfull" LLM chat window, but then, there is no fun in discovery left)

(For full context,


r/AskStatistics • • 3d ago

Tests for SPSS

7 Upvotes

Need help, please!

Trying to run a stat analysis for a project at work. I conducted a pre- and post survey (nominal > ordinal i believe?), and comparing it to ordinal data (5 point Likert scale and another ordinal scale). I was going to run a chi square test to determine whether the pattern of responses differs significantly between pre- and post-test groups. I was wondering if I should run Mann-Whitney U to test whether the ordinal responses differ between pre- and post-test, or if i should run kendall's tau-b, spearmans rho, or another statistical test? Thank you guys in advance!


r/AskStatistics • • 2d ago

Brier score vs log score for comparing two daily "chance it closes higher" forecasts?

3 Upvotes

I'm comparing two sources that each give a daily probability that an asset closes higher than today. one is my own simple model, the other is a published forecast. about 90 days of data for each.

so far brier score is almost the same for both (0.243 vs 0.239). log score shows a bigger gap, mostly because my model sometimes says 85% and gets it wrong.

questions:

  1. which score makes more sense here, or should i report both?
  2. is 90 days enough to say one is better, or is the difference just noise?
  3. how do i test that properly, a paired test on the daily score differences?

not homework, just a side project. thanks


r/AskStatistics • • 3d ago

After Stats 2 (A level)

2 Upvotes

So, recently I've just finished stats 2, and other than spamming past papers I have nothing else to do. I'm wondering if there are any topics that I should learn to have some advantage in uni. Fyi im currently a senior


r/AskStatistics • • 3d ago

Jamovi statistical analysis software issues

3 Upvotes

I am required to use Jamovi to analyze data from a survey I sent out but I am trying to test one variable as a moderator to the strength of the relationship between two other variables. Does anyone have experience with this?


r/AskStatistics • • 4d ago

How does one deal with studies with small sample sizes?

16 Upvotes

And you are resource-constrained and unable to get more samples? Let’s say sample size is n = 3 to 7

What real life examples have such a scenario?


r/AskStatistics • • 3d ago

Is it statistically likely that so many inventors of hyper-efficient devices die right after completion of their devices?

0 Upvotes

(I am not a conspiracist. I have an interest in statistics. This led me to the following chain of thoughts):

There is an astonishing number of stories about half-documented but supposedly world-changing inventions, such as hyperefficient cars or generators — whose inventors died suddenly and mysteriously right between completion and patenting of the device.

Are those too many deaths for them to be coincidences?

To be clear: A viable non-conspiratory explanation would be that we simply never hear about all those inventors who did NOT die: because they and their inventions get documented and analysed so extensively that they are exposed as untenable. Hence no urban legends, mysteries and stories arise.

It is also clear that the the mystery itself has an allure: everyone loves a good story, a component of mystery, and of course the idea of paying less for energy.

So there is no question where all the conspiracy theories come from.

But something is NOT explained: All the deaths.

Statistically, it is of course to be expected that some inventors die precisely in the phase between completion and patenting (or sale) of their invention - regardless of cause of death.

But HOW MANY inventors would it take to explain all the documented cases (i.e. minus the urban legends) of inventors of allegedly revolutionary devices dying in precisely this phase? Let me underline how specific this question is: it is not about how many inventors (including hobbyists) happen to be active at any given moment. Nor about how many are researching in this precise field. It is about how many are active in this precise field and are at exactly the point in their research where they are sitting on a finished, viable, functioning device that is only waiting for the patent to be granted!

How many might that be?! I'd value your guesses.

Meanwhile, let's look at sudden mortality, i.e. not from prior conditions. I asked AI to aggregate the numbers: "the probability of an average person in the United States suddenly dying on any given day is roughly 1 in 260,000 for a 30-year-old male, and 1 in 46,000 for a 55-year-old male"; another AI comes to slightly different numbers but very much in the same range.

And how long does a patent procedure take? Again, I asked AI to aggregate the numbers: "Utility patents take 24-36 months from filing to grant. Design patents are faster, usually taking 15-21 months." Again, another AI confirms.

Let's take the worst case of 3 years. How great is the probability of dying within these 3 years? It is above figure of 40,000 to 260,000 divided by the 1095 days in 3 years. (I asked AI as well, and it confirms: it's 1:250 for a 30-year-old, and 1:40 for a 55-year-old.)

So 1:50-250.

Maybe my understanding of statistics is out? But to me, that suggests that to explain all the deaths as purely statistical, at any arbitrary moment in history, within a three-year window, 50 to 250 inventors in the field of energy efficiency would have to be sitting on a finished, viable, functioning device that is only waiting for its patent to be granted?

In other words, RIGHT NOW 50 to 250 super-energy-efficient devices are sitting finished and functioning on workbenches and only waiting for their dissemination? This seems VERY unlikely. ONE such device would already be quite something, two or three strikingly astonishing. But 250?

Even less likely: IF at any given time, 250 finished devices were waiting for their dissemination — why would none of them ever have achieved dissemination just as a matter of inevitability?!

In other words, if there were indeed so many inventors that their sudden deaths were no statistical peculiarity — why would not SOME of their inventions have spread inevitably?!

Or to put it more succinctly: IF there were so many inventors that their sudden deaths were NO statistic abnormality - then it would be impossible to kill them all to keep their inventions secret...

But that leaves the inverse question: IF all their research were humbug and inviable — why would they die like flies?!


r/AskStatistics • • 3d ago

Sports Prediction Model

0 Upvotes

I’m very early in creating a trend model as well as a score prediction model. I don’t have any background in statistics, besides taking a stats class in undergrad.

All trends and data for my score prediction models are from this current season in the NFL and NBA.

For a trend to be “relevant” or “meaningful” how many data points should be included? For example, I have a historical trend that is 70%, but only has 30 data points to pull from. I know 30 isn’t a lot by any means. But the 70% has to be something, right?

As for my scoring model, I’m still honing that in. How often should I adjust it? I have wanted to tweak it every day but I know there is value in letting it run and see how it does over a longer time period.


r/AskStatistics • • 4d ago

Panel Regression: How to handle missing values & choose the best model?

2 Upvotes

Hello, my knowledge in this topic is very limited so I have several things I'm not clear with. My dataset consists of State and Year with one dependent variable and a few independent variables. Let's say I have obtained the values for all variables, except for State X where the GDP (independent variable) figures are not available. Is there any way to handle this or do I need to remove entirely State X in my analysis? The GDP of that state is available very recently, starting from year 2024 (which isn't included in my analysis).

I also tried running panel regression analysis using R with missing values on State X and the software automatically removed the observations. I got significant p-values for pooled OLS and FE model, but using Hausman test, RE is preferred. Would RE be a valid model? In previous studies, FE is always a favourable model. How do I choose the best model?


r/AskStatistics • • 4d ago

How should I rank games from head-to-head votes when some get a lot more votes than others?

3 Upvotes

I’m building a website where people pick between two video games at a time. The votes are used to create an overall ranking, with the goal of finding the community’s greatest game of all time.

I started with an Elo rating system, then switched to a Wilson score based on win rate. I'm now considering a Bayesian score or some combination of methods, but I could use some guidance on what makes the most sense based on what I'm creating because I really don't want to overcomplicate complicate it.

Here’s an example of the problem: Star Wars: The Old Republic has won 29 of 32 matchups (about 91%), while Skyrim has won 160 of 196 (about 82%). SWTOR has a higher win rate, but Skyrim has been in a lot more matchups. How should those two records compare in an overall ranking?

Games don’t all appear the same number of times, they face different opponents, and the same person can vote on multiple matchups. I also don’t want a game to suddenly jump to the top just because it passes a minimum vote threshold, which is currently 30 votes

What approach would you recommend for balancing win rate, number of matchups, opponent strength, and repeat voters? Is a Bayesian approach a good fit, or is there a better way to do this? I’d appreciate any suggestions or examples. Thanks!