[cherdano]

Just as a P.S. on this discussion - unfortunately relevant, because unfortunately Henaghan is rather influential within the covid-19 discussion in the UK:

Cyberyeti, on 2020-November-02, 15:31, said:

Another article I came across suggesting the new UK lockdown is based on poor modeling https://www.cebm.net...1pW9bRJjbzCRxo4 interested in hearing comments.

By now, the rolling 7-day average around Nov 1 for deaths (by date of death, not date of report of death) is 329.4 - not 200 as Henaghan claims.
This should have been obvious to him, as a professor essentially for statistics. That it wasn't either means either he is intentionally misleading, or his bias against lockdowns/restrictions means he can't quite think clearly, not even about something as simple as number of deaths. Indeed, swap 200 for 330 in the following two paragraphs, and suddenly the whole point of the article becomes rather empty:

Quote

The PHE/Cambridge model on this date has an estimated projection of 1,000 deaths, yet deaths are averaging just over 200 deaths for this date on the government’s staging website.

We estimate the Imperial model to be projecting approximately 486 deaths on the 1st of November, LSHTM 266 deaths and Warwick 234.

So yes in my ideal world we'd stop paying attention to him.

[barmar]

Another promising vaccine announcement came out today. Moderna announce a vaccine that's at least 94% effective. It still has to go through additional safety testing, like the Pfizer vaccine announced last week. But it sounds like it may be less expensive to distribute, as it can be stored in a refrigerator for 30 days, rather than requiring extreme cooling.

[cherdano]

The 90% vs 94% is a bit of a red herring as the confidence intervals certainly overlap, and as Pfizer wasn't precise on the observed efficacy in their trial results.
The bigger difference is that Moderna told us more detail, and it's all very good. There were no severe cases in the vaccine group (compared to 11 in the placebo group). It seemed to work just as well among elderly (though of course that is based on very small data), and ethnic minorities. They also gave more details on side effects, which seemed harmless.

[pilowsky]

Perhaps the statisticians can clear something up for me here.
Moderna reported that they undertook a trial with 15000 people in a placebo arm and 15000 in a treatment arm. Both received 2 shots. There were minimal adverse effects.
The 94.7% efficacy that they are talking about is that 5 people in the treatment group got COVID which was mild. and 90 people in the placebo group got COVID, some severely.
It is worrying to me when people start talking about percentages when dealing with numbers smaller than 100.
So what they mean is that of all the people got exposed to COVID, the vaccine was effective in 94.5% of cases.
In fact, it may be much better or worse since we don't know how careful/ careless the participants were.
Simply being enrolled in a blood pressure trial will often lower your blood pressure.
Still, in terms of efficacy, it really does look like a slam-dunk.

Clearly, the best news is that this vaccine is stable at higher temperatures.
Less good is that we do not know yet about the efficacy in people that are older or sicker or who are taking various medications. Of course, it definitely will not work in people that don't take it. That's not a joke.

I'm going to wait for a bit more information before I get too excited.
The Pfizer CEO just sold 60% of his stock.
So did the Moderna executives (albeit some time ago).
This is not a good look.

[cherdano]

cherdano, on 2020-November-16, 19:17, said:

The 90% vs 94% is a bit of a red herring as the confidence intervals certainly overlap, and as Pfizer wasn't precise on the observed efficacy in their trial results.
The bigger difference is that Moderna told us more detail, and it's all very good. There were no severe cases in the vaccine group (compared to 11 in the placebo group). It seemed to work just as well among elderly (though of course that is based on very small data), and ethnic minorities. They also gave more details on side effects, which seemed harmless.

And Pfizer follows up with more details now that their trial has observed as many cases as it was originally set out to be. Not surprising to anyone who actually read their first press release that the observed efficacy was actually higher than 90% - similar to Moderna, 95%. And essentially the same efficacy among older participants, and consistent across other subgroups (gender, ethnic minorities). One out of ten severe cases in the vaccine group.

All great news. But for most of us, things will change only very slowly. I.e., our own risk will hardly change until vaccines become more widely available; still it'll be nice to know that fewer people are dying!

[cherdano]

pilowsky, on 2020-November-17, 02:13, said:

It is worrying to me when people start talking about percentages when dealing with numbers smaller than 100.

Does it worry you when I tell you that there is a 16.66666...% chance of rolling a six with your next dice throw?

Quote

So what they mean is that of all the people got exposed to COVID, the vaccine was effective in 94.5% of cases.
In fact, it may be much better or worse since we don't know how careful/ careless the participants were.
Simply being enrolled in a blood pressure trial will often lower your blood pressure.

Uhm, that's why the trial was double-blind - the participants wouldn't know whether they got the vaccine or the placebo. (Unless they could guess from the side effects - but that would mean those with the placebo would be more cautious, i.e. the actual efficacy would be even bigger than observed.)

Given case numbers in the UK and the US, I suppose we can look forward to other vaccine trial reporting soon as well.

[pilowsky]

cherdano, on 2020-November-18, 06:24, said:

Does it worry you when I tell you that there is a 16.66666...% chance of rolling a six with your next dice throw?

Uhm, that's why the trial was double-blind - the participants wouldn't know whether they got the vaccine or the placebo. (Unless they could guess from the side effects - but that would mean those with the placebo would be more cautious, i.e. the actual efficacy would be even bigger than observed.)

Given case numbers in the UK and the US, I suppose we can look forward to other vaccine trial reporting soon as well.

Yes, it does worry me when you say there a 16% chance etc, and I think that you know why. What you are doing, metaphorically speaking is projecting beyond the available data.
We are not talking about a single 'die' and people are actually dying. So, it's not just a matter of rolling the dice if I can be light-hearted and mix my metaphors.

If 2 people are sick and both are given an effective drug and only one gets better, can we claim that the drug has only a 50% success rate? should we now take it off the market because it doesn't work?

Of course not. This is really simple stuff. You CAN turn any ratio into a percentage, that doesn't mean that it's a good idea, because if the numbers are small it's misleading. Here, the numbers are in excess of 10:1 so I'm happy.

I would prefer it if you said 1 in 6.

As to the second point, even though the trial was double-blind, the trial was still a trial. I assume that you have no experience with clinical trials. If you did, you would know that trial participants are a very special breed of highly motivated altruistic people.
I suspect that is one reason for the apparently very low rate of infections. In the placebo group the positivity rate was .006% (90/15000). That seems like a pretty effective treatment don't you think?
Trial participants are very altruistic, highly motivated and very health conscious.
They are not the same as members of the general population.
They get much better health care than other people.
Trial participants are also ordinary people, they get 'side-effects' just from being in the world. Sometimes the side-effects occur in the trial arm, sometimes not, sometimes they are serious, sometimes not.
Fortunately, trial participants no longer include prisoners etc, although in some countries where there is a history of human rights abuses they may well be using unwilling participants.

China for example is known for this.
Ireland is threatening to forcibly repatriate healthcare workers without citizenship now that the pandemic is waning.
Stalin forced slaves onto the battlefield. I have no idea what is happening ATM.
Americans and British and everyone else in the world enjoy the use of slave and child labour from other countries.


It's nothing personal, I am just a stickler for accuracy and precision. I see things as they are. When I get things wrong I admit it. I don't care.

[Zelandakh]

Helene, this would be a really good time for you to chime in and explain confidence intervals for non-mathematicians.

[pilowsky]

Zelandakh, on 2020-November-18, 15:49, said:

Helene, this would be a really good time for you to chime in and explain confidence intervals for non-mathematicians.

I would also value Helene's opinion.
I would also like to know the relevance of a clinical trial where 95 people were infected out 30,000 participants. This seems to be an astonishingly low infectivity rate. Or am I missing something? I really hope that I have missed something. I just missed 4S= a moment ago so it wouldn't surprise me.
The relationship of these data to the real world is yet to be determined as we say in the res. biz.

[hrothgar]

pilowsky, on 2020-November-18, 16:49, said:

I would also value Helene's opinion.
I would also like to know the relevance of a clinical trial where 95 people were infected out 30,000 participants. This seems to be an astonishingly low infectivity rate. Or am I missing something? I really hope that I have missed something. I just missed 4S= a moment ago so it wouldn't surprise me.
The relationship of these data to the real world is yet to be determined as we say in the res. biz.

My understand of the Moderna results is that they enrolled 30K people in a trial.
Some of the folks in the trial received a placebo, others got the actual vaccine.

Of the folks who were enrolled in the trial, 90 of them ended up getting Coronavirus.
So, 1:333 participants in the trial contracted COVID

When they went and looked at who precisely contracted COVID, they found that

85 of the people who contracted COVID received the Placebo
5 of the people who contracted COVID received the vaccine

85/90 = 94.4%

The key issue here is the relative small number of individuals who ended up contracting COVID regardless of whether or not the received the vaccine.

With only 1 every 333 participants contracting the disease, you need an enormous sample size to boost those numbers significantly.

Or, alternatively, you need to start vaccinating people and then deliberately exposing them to COVID.
(However, this is an incredibly dicey subject which raised all sorts of moral quandaries)

One good thing is that the first two vaccines were both using similar techniques (mRNA) and both are showing consistent results...

[cherdano]

pilowsky, on 2020-November-18, 16:49, said:

I would also value Helene's opinion.
I would also like to know the relevance of a clinical trial where 95 people were infected out 30,000 participants. This seems to be an astonishingly low infectivity rate. Or am I missing something?

Yes you have missed something. Why is that rate low? Until a few weeks ago, a rate of 50 infections for a population of 100,000 within a week was considered high... So if the participants had received their second dose on average near the end of August, you'd be now have about 10 weeks of data, for expected 500 cases among 100,000, or 75 for the placebo group of 15,000. The only reason Pfizer already has 160 cases in the placebo group is that infections have risen way beyond that level in recent weeks.

Whether the 160 cases come from 15,000 study participants or 300 changes the distributions a little bit, but that really only makes a minor difference. If efficacy for symptomatic infections was all there is to it, we'd have stopped the study long ago - indeed, Pfizer originally had n interim point where they'd look at the date after 32 total cases. And a split of 6:26 in the vaccine versus placebo group would have been deemed enough to reject the hypothesis that the vaccine is not effective.
They published the entire protocol - which basically includes an advance commitment on how they will interpret the data - beforehand: https://pfe-pfizerco...al_Protocol.pdf

[helene_t]

Zelandakh, on 2020-November-18, 15:49, said:

Helene, this would be a really good time for you to chime in and explain confidence intervals for non-mathematicians.

So in Pfizer/BioNTech's preliminary trial, presumably 8/23000 vaccinated people got Covid symptoms while 86/23000 non-vaccinated people got Covid symptoms.

We say the point estimate of the vaccine efficacy was 78/86=0.91 because, if there were no statistical flukes that worked against or in favour of the vaccine, 86 vaccinated people would have gotten Covid if not vaccinated, and apparently 78 of them were protected by the vaccine. You can then raise all kind of objections like Covid symptoms being masked by vaccine side effects or the vaccine just delaying, not suppressing, the symptoms, but let's for the sake of the argument assume the numbers are correct and relevant.

But BioNTech could of course just have been lucky that few people in the vaccine group were exposed to the virus, just like you can throw three sixes in yatzee at your first go even if statistically you "should" throw only one. The confidence interval tells us what the true efficacy could plausible have been.

Strictly speaking, the efficacy could be anywhere between 0% and 100% (or even less than 0%, I'm not sure how we would talk about the efficacy of a vaccine that is worse than placebo). Just like card shuffling machine that gives you a hand of thirteen spades 20 times in a row could in theory be a fair shuffling machine. But we want to report an interval which by some definition covers all efficacy values that a reasonably consistent with the data.

If you suggest that the true efficacy may have been 100% I will tell you that that's impossible since some people actually got Covid despite getting the vaccine. What about 99.999% then? Could be, but if the efficacy was really 99.999% then the probability of getting something at least as surprising as the 86/8 split would be very low, so we can rule that out.

We can't "prove" that the true efficacy isn't 99.999% but then again, nothing can be proven except in pure mathematics. If the observed data would be extremely unlikely given some hypothesis, it seems reasonable to reject the hypothesis.

It turns out that the confidence interval for the efficacy is 82%:96%. This means that if you suggest that the true efficacy was anywhere outside that range, say 97% or 80%, you would find the observed 8/86 split quite surprising: there would be less than 5% probability of observing such a surprising split, and you are therefore inclined to say that an efficacy of 97% or 80% is not really consistent with the data.

Maybe you would say that 5% probability doesn't count as "very surprising", and you could set the threshold at 1% instead, which would give you a slightly wider confidence interval of 78%:97%.

In the study sample, the efficacy was really 91%, assuming that the data were correct. The confidence interval doesn't dispute that. What it says is what we can infer about the data generating process.

We would like to think that we can therefore assume that when the vaccine is given to future patients, its efficacy will be in the 82%:96% range. This rests on the assumption that future patient outcomes are generated by the same data generating process as the patients that were in the trial. Of course, this will never be completely true: future patients may be exposed to different strains of the virus, may have their outcomes accessed differently etc. So while "in theory" we would expect that only 5% of approved drugs turn out to have a post-approval efficacy outside the 95% confidence interval reported in the pre-approval trial, in reality it's a bit more than 5%.

Slides from a recent talk I gave about confidence intervals:
https://drive.google...iew?usp=sharing

[cherdano]

Helene, one update. In the press release from today, Pfizer/BioNTech specified that it was 8 cases (of symptomatic covid-19) in the vaccine group, and 162 in the placebo group. In the earlier press release from last week, they hadn't given exact numbers, just stated that after 95 cases, the trial indicated an efficacy of > 90%. So they probably had fewer than 8 cases in the vaccine group then.

[johnu]

How the Out-of-Control Pandemic Is Speeding the Testing of Vaccines

Quote

The surging virus has already allowed Pfizer and Moderna to accelerate the testing of their vaccines, which appear to be very effective at preventing Covid-19.

And if, as seems inevitable, the virus continues to proliferate — it is spreading faster than ever in the United States and some other countries — it is likely to speed the evaluations of promising vaccine candidates from other pharmaceutical companies.

The scientific principle behind this is simple. Researchers can't test the real world effectiveness of a vaccine if there is no disease present. It wouldn't make sense to test vaccines in countries like Taiwan or Vietnam because it could take years before you had enough data to make decisions, too late to be of much use in the current crisis.

With the totally out of control spread of the Trump Virus, the US is the world's best place to test vaccines because case data is piling up at record speeds. Certainly a very dubious achievement and nothing to be proud of, but at least the US is helping the rest of the world by filling up its morgues and ICU's, thanks to the Manchurian President's policy of Herd "Mentality" (sic)

[thepossum]

EDIT Still being edited. Apologies. Too many things happening at once

I haven't had chance to read any studies recently, I'm rather tired and a bit rusty and any comment I make would be a personal view and nothing else. But I'm seriously concerned about the way this is being played out in the media. And without even thinking or analysing anything comparing 90/15000 with 5/15000 wouldn't fill me with any excitement. Thats of course just thinking about the those low figures crudely and without any further information which could make them look more significant. Who knows about the details of the sample and the cases etc. It would take much more reading and understanding of the study.

The fact insiders have been selling shares on early market excitement is interesting too

Another point of personal cocnern I have (in terms of practicality and justice) is any vaccine requiring -80(??) degree refigeration. Maybe in certain specialised environments but from a global response perspective it concerns me a bit

EDIT Apologies getting two trials mixed up in the chat so hope the following two articles clarify.

Here is a recent article about Moderna in Nature Moderna Nature Article

Note, just reading an article in Nature saying the Pfizer trial needs to get to 164 cases overall for results to start to be more exciting (my words)

[cherdano]

The Pfizer trial is already beyond 164 cases. But 164 cases was deemed the appropriate number to prove effectiveness of a vaccine that works 60% of the time. For the observed efficacy, a much lower number of cases would have been enough.

Just yesterday, the WHO announced the end of an Ebola outbreak in Congo, thanks to a vaccination campaign. The vaccine - needs to be stored at -80C.

[thepossum]

cherdano, on 2020-November-18, 20:44, said:

The Pfizer trial is already beyond 164 cases. But 164 cases was deemed the appropriate number to prove effectiveness of a vaccine that works 60% of the time. For the observed efficacy, a much lower number of cases would have been enough.

Just yesterday, the WHO announced the end of an Ebola outbreak in Congo, thanks to a vaccination campaign. The vaccine - needs to be stored at -80C.

Thx. As I said, I just found the Nature article and was checking on the numbers related to above discussions

Relating to Ebola though, we are not talking about vaccinating a whole planet in quick time, so I imagine it has proved practical to have the mobile refrigerated units (or however it is done) for relatively small outbreaks of a very different virus type

[pilowsky]

hrothgar, on 2020-November-18, 17:25, said:

My understand of the Moderna results is that they enrolled 30K people in a trial.
Some of the folks in the trial received a placebo, others got the actual vaccine.

Of the folks who were enrolled in the trial, 90 of them ended up getting Coronavirus.
So, 1:333 participants in the trial contracted COVID

When they went and looked at who precisely contracted COVID, they found that

85 of the people who contracted COVID received the Placebo
5 of the people who contracted COVID received the vaccine

85/90 = 94.4%

The key issue here is the relative small number of individuals who ended up contracting COVID regardless of whether or not the received the vaccine.

With only 1 every 333 participants contracting the disease, you need an enormous sample size to boost those numbers significantly.

Or, alternatively, you need to start vaccinating people and then deliberately exposing them to COVID.
(However, this is an incredibly dicey subject which raised all sorts of moral quandaries)

One good thing is that the first two vaccines were both using similar techniques (mRNA) and both are showing consistent results...

Well, there's the rub,
Since coronavirus is the agent and COVID is the disease, and the participants in the trial are a quite different population to the people that are out there in the 'real world' where Doctors treat patients, that's where the rubber hits the road.

Thank you all for clarifying the maths.

Following the (media) commentary, there is now a major structural problem getting the vaccine to people who need it for many reasons, and then getting them to take it.
Why is compliance a problem? Just to give an example, cigarette smoking causes 480,000 deaths every day in the USA. Alcohol causes 95,000 and so on. One of the main reasons that COVID-related deaths have hit the USA and the UK so hard is a failure to 'believe' in it.
Along with pre-existing morbidities many of which are self-inflicted (diet/cigarettes).
Confidence intervals can't overcome this.

It may seem that I am changing the topic, but it does as I say relate to the original highly motivated study participant population.

Speaking of which, does anyone know how many of the participants completed the study? 15000/15000 is an amazing result for a clinical trial. One normally loses at least one or two people to follow-up.

[barmar]

I'm not a statistics expert, but I think the real point is that developing a COVID-19 vaccine is extremely urgent. Normally it takes several years to develop and test vaccines, this allows you to wait for many results and have much more confidence in the results. But we need something ASAP, we can't wait for thousands of trial participants to die.

But there's also more to it. This is Bayesian. We have prior expectations based on the theory behind the vaccine design, and 90 vs 5 is significant enough to confirm the hypoethesis, even though they're both small compared to the group sizes of 15000. It's unlikely to be a statistical fluke since we have good reason to believe that the vaccine will be effective.

As an analogy, suppose you're trying to determine if a die is fair. Ideally you would count the number of all the different faces, and see if they're approximately equal. But if you're already pretty sure that it's a fair die, and you just want confirmation, you could get away with counting just one of the faces. If it's close to the expected fraction of rolls, you'd consider your hypothesis confirmed. Not as well as the first method, but good enough.

[pilowsky]

That point is completely true. It really does look like an effective vaccine in the group that it was tested on.
Let's hope that it also works when it gets distributed, that the supply chain works, that people that get very big doses of the virus are also protected and so on.

It is extremely promising. The data was analysed appropriately. It is a good start.

None of that means we stop asking questions, thinking or taking precautions.

Look how difficult it was to eradicate polio, TB is still a massive problem. To say nothing of malaria etc etc.

I get it, it's a Bridge Forum so there is a natural tendency to focus on probability and restricted choice (Bayes theorem) which is where my research career started.
I'm just saying that there is much more to it. Developing a vaccine is important, but it is not the most important.

Just compare the experiences of different countries around the world.