A reaction to the report "Systematic literature analysis and international comparison 'Excess Mortality'", Cochrane Netherlands, April 3, 2023

Authors (in alphabetical order): Bram Bakker, Bastiaan Geelhoed, Maurice de Hond, Jillis Kriek, Fritsander Lahr, Ronald Meester, André Redert, Herman Steigstra, Anton Theunissen

Also accessible via Ronald Meester’s LinkedIn page.

Date: April 24, 2023

Introduction and context

On 14 April 2023, the Dutch Minister of Health, Welfare and Sport, Ernst Kuipers sent the above report[1] to the House of Representatives. This gave the report a status that justifies a response. The report concerns a meta-analysis, which describes or compares several international studies on excess mortality. In principle, meta-analyses are useful, although conclusions are often difficult or impossible to formulate, precisely because of the fact that causality is extremely difficult to establish in an epidemiological setting of this size.

In the accompanying letter to the House of Representatives, Kuipers provides the following summary of the results:

“The study describes the emergence of a trend of higher excess mortality (the difference between observed and expected mortality (per 1 million population)) in Eastern European countries and lower excess mortality in Scandinavian countries, Australia and New Zealand. In addition, the study also includes a review of COVID-19 mortality. The highest COVID-19 mortality (per 1 million population) was observed in Western European countries, while COVID-19 mortality was also the lowest in Australia and New Zealand. The differences in the trends of excess mortality and specific COVID-19 mortality could possibly be explained by differences in the reporting of COVID-19-related deaths as well as differences in the ability to test individuals for COVID-19. The researchers also indicate that international differences in excess mortality may be explained by measures against coronavirus, organisation of and access to care, population characteristics, behaviour, geographical location and differences between countries in data quality. Finally, the researchers point out that a number of studies show that excess mortality is lower among the population of fully vaccinated people.”

There is much to be said about this quote, and in principle it is good that the uncertainties and difficulties in comparing studies are mentioned. However, the last comment in particular is rather misleading because it suggests causality. Because what does the report say? We quote on page 8: “Vaccinations appear to be associated with lower excess mortality, but since higher vaccination rates were associated with the emergence of the Omikron variant, no causal relationship can be established.” Cochrane’s wording thus paints a different picture from the minister’s suggestion. And that’s not all: as we will explain below, no conclusion about a relationship between vaccination rate and the level of excess mortality can be drawn from the report.

Of course, we are not just concerned with how Kuipers informs the House of Representatives. Rather, we make some observations about the report itself. We first present a list of some comments on the report. Our list of points of attention (which we present below) is by no means exhaustive – in fact, that would require a much greater investment of time. Subsequently we show that the Cochrane report uses and cites sources incorrectly. We end with a brief conclusion.

Some comments on the report

The period 2020-2021 was probably chosen because of the Omtzigt motion. Since then, however, a wealth of information has emerged, in particular on the side effects and damage of vaccines and on the consequences of measures such as lockdowns. It is in fact the excess mortality that persists post vaccination that worries us, and which did not disappear after the milder Omicron variant took over. Claims that may have seemed plausible until 2021 do not hold up in the light of current insights. Consequently, the report is already outdated when published and therefore of little or no use.
It’s nearly impossible to compare studies in which different methodologies are used. No systematic methodological assessment of the various studies is given in the report. It is not clear what system was actually utilised in selecting and assessing the various studies. Hence, the study isn’t very systematic at all. If one doesn’t look very deeply into the methodological strengths and weaknesses of individual studies and one doesn’t think very carefully and deeply about them, including all the pitfalls and biases, and one just says that “several studies state this”, then a meta study isn’t really useful.
Even the expression “excess mortality” itself has different definitions in different studies. [2] That in itself should be enough reason not to draw any conclusions from this report.
On page 6, the report states that excess mortality has increased in Eastern Europe. This is completely incorrect, see Excess mortality in 34 countries, failing vaccines? (virusvaria.nl) for a discussion on this claim, which can only result from a naive and incomplete understanding of the data. Furthermore, countries like Belarus, Iceland and Greenland are not suitable for a this kind of study, for various reasons. That analysis also shows that what was mistaken for a correlation between vaccination rate and excess mortality cannot simply be that, based on historical data.
On page 7, we read that no studies were found in which vaccination and excess mortality were positively correlated. That can only mean that those studies were not included. Such studies do exist – in fact, there are many.[3]Indeed, here we see that the report was already outdated the moment it was published.
It is completely unclear how the various sources of bias, as for example recently described in Sources of bias in observational studies of covid-19 vaccine effectiveness – PubMed (nih.gov), were dealt with in the various studies as well as in this report.
Cause of death certificates from at least CBS (the Dutch office for national statistics, red.) are shrouded in mystery, for various reasons. [4] For example, the WHO prescribes that when covid is listed somewhere on the death certificate, “Covid-19” must be recorded as the leading cause of death. There are many reasons why these data are not suitable as input for scientific research.
It has become clear that there are large differences between batches of vaccines. In Denmark, about 70% of adverse events were found to come from 4% of the batches; a strong indication of a relationship between vaccines and mortality. [5] This may make a meta-analysis rather pointless.
There are a lot of omissions in this report that are very relevant to the discussion on excess mortality, for instance with regard to the remarkably short protection against infection by the vaccines and the increasingly short protection by boosters. But we also mention the disappointing protection of the vaccines against new virus variants (a possibly important partial explanation of why the Omicron period was associated with high levels of excess mortality), the now well-known side effects of the vaccines in the literature, which are quite worrying, the now well known differences between the various vaccine types and brands, and the striking over-mortality rates in the second half of 2021, all of 2022, and early 2023 in many highly vaccinated countries.
The exclusion criteria on pages 11 and 12 are not compatible with research questions 5 and 6.
On page 13, we read, “In this way, different countries could be compared, and differences in follow-up duration between studies was corrected for.” The differences in follow-up duration are corrected by Cochrane by dividing the relevant variables (mortality/excess mortality etc.) by the follow-up duration, so they were presented as rate (mortality per year). If a publication only focusses on one wave, that wave gets hugely magnified when the values are converted to rates. A distinction should be made to individual waves. For each wave, you can convert values to rates so that countries can be compared. But now there is even the danger of comparing wave x in country A with wave y in country B on the basis of mortality rates, while in the meantime vaccinations may even have been rolled out.
For research question 5 (international differences), only studies that presented data from more than 5 countries were selected. However, even if studies cover only 1 country, these studies can still be used to compare countries, provided appropriate normalisations and standardisations are done. Thus, we think that by applying this selection criterion, there is an unnecessarily restrictive selection of studies.
Finally, we also note that there is an interesting difference between the version of the report sent to the House of Representatives and the version found on ZonMw’s site[6 ] (the client, red). On page 45, the ZonMw version reads: “Differences between excess mortality and COVID-19 mortality may be caused by different data quality.” This sentence is not in the House of Representatives version. We find it confusing that there are apparently different versions in circulation, especially when it comes to remarks that matter. Needless to say this raises questions.

The use of resources

As a sample, we checked whether the Cochrane report used all sources correctly. This turned out not to be the case at all, and we illustrate this with some examples:

Example I (page 41)

“One study presented excess mortality in relation to vaccination coverage for four regions in the United States and for three different time periods (Table 11 and Excel Appendix). [62] Excess mortality was the highest for all regions in the period with the lowest vaccination rate of about 7%.“

The publication being cited is (we adhere to the numbering from the Cochrane report) [62]Stoto MA, Schlageter S, Kraemer JD. COVID-19 mortality in the United States: It’s been two Americas from the start. PLoS ONE 2022; 17(April 4) doi:https://dx.doi.org/10.1371/journal.pone.0265053.

The statement in the Cochrane report is incorrect because it was probably overlooked the fact that there were periods when the vaccination rate was not reported (empty cell in the table), but in all likelihood must have been lower than in the other periods in the table when the vaccination rate was reported (because vaccination rates increase over time). We see, for example, thatthe Northeast region has the lowest excess mortality (0.111) per capita in the period 31 May 2020 – 3 October 2020, while the vaccination rate for that period was not mentioned, but was probably substantially lower than 7%, because mass vaccinations had not yet started then (so it will be 0% or at least close to 0%). This would indicate the opposite conclusion (than the one in the Cochrane report). Thus, the Cochrane report suggests a conclusion (that lower vaccination rate leads to higher excess mortality) that is not made by the authors of the article. Table 11 in the Cochrane report thus quotes selectively.

Example II (page 42)

The Cochrane report continues on page 42 with: “Seven studies described the relationship between excess mortality and vaccinations. Multiple studies showed that increased vaccine availability and usage correlated with lower excess mortality (Appendix 11B). [17, 31, 62, 70, 73]“.

From reference [62] we have already shown that the Cochrane report does not interpret it properly. Do the other references show the correlation that the authors of the Cochrane report point out? We’ll check.

[17]Bell E, Brassel S, Oliver E, et al. Estimates of the Global Burden of COVID-19 and the Value of Broad and Equitable Access to COVID-19 Vaccines. Vaccine 2022; 10(8) doi:https://dx.doi.org/10.3390/vaccines10081320

The conclusion of the Bell paper does not state that increased availability and usage of vaccines have a correlation with lower excess mortality. It may be that the authors of the Cochrane report came to their conclusion based on the following statement in the Bell paper: “Our model’s results under Scenario 1 show that in 2021 1.4 million direct deaths, 4.3 million excess (direct and indirect) deaths and 6.0 million hospitalisations have been averted and hospital resources worth USD 59 billion have been saved by COVID-19 vaccinations.” Regardless of whether this conclusion is correct (after all, it is based on a model/scenario), it is simply something different from what is claimed in the Cochrane report.

What about Source [31] then? This is aboutFazekas-Pongor V, Szarvas Z, Nagy ND, et al. Different patterns of excess all-cause mortality by age and sex in Hungary during the 2nd and 3rd waves of the COVID-19 pandemic. GeroScience 2022 Doi:https://dx.doi.org/10.1007/s11357-022-00622-3.

This study does not at all unambiguously show that (in the words of the Cochrane report) “increased vaccine availability and use would correlate with lower excess mortality“. It shows a small decrease in “excess absolute mortality” from the second to the third wave while simultaneously increasing vaccination rate. But this effect, the authors of the Fazekas paper also broke down this effect by age, and it seems to be driven mainly by the group aged 65 and above. The age groups 35 – 44 years, 45 – 54 years and 55 – 64 years all three showed an increase in excess absolute mortality, which was accompanied by a similarly increasing vaccination rate.

Source [70] then? That is Watson OJ, Barnsley G, Toor J, et al. Global impact of the first year of COVID-19 vaccination: a mathematical modelling study. The Lancet Infectious Diseases 2022; 22(9):1293-302. Doi:https://dx.doi.org/10.1016/S1473-3099%2822%2900320-6.

This is a mathematical modelling of how COVID-19 vaccination can reduce excess mortality. The model does not take into account any deaths due to vaccination, adverse events or vaccine (in)safety. While interesting, this study should not be used to draw conclusions regarding mortality due to vaccination, adverse events or (in)safety of the vaccines, as the model used assumes a priori that these aspects play no role.

Source [73] then? This is about Zhou F, Hu TJ, Zhang XY, et al. The association of intensity and duration of non-pharmacological interventions and implementation of vaccination with COVID-19 infection, death, and excess mortality: Natural experiment in 22 European countries. Journal of Infection and Public Health 2022; 15(5):499-507. Doi:https://dx.doi.org/10.1016/j.jiph.2022.03.011.

This publication shows the opposite effect to what is claimed in the Cochrane report. In Figure 35, it clearly shows that the “vaccine effect” on excess mortality increases with increasing “vaccination rate”. That the relative risks (RR) are below 1 is irrelevant here. What matters for the correlation between excess mortality and vaccination rate is only the fact that the RR’s increase with increasing vaccination rate.

Discussion and conclusion

We did not have time to look into all referenced sources of the Cochrane report, but in those that we did look into, we found consistently problems. We have encountered many more problems than those we have mentioned here. Of course, the question is how this is possible, but we’re unable to answer that question.

For context, it might be worth noting that Cochrane’s Dutch team, including the director of Cochrane Netherlands, consists exclusively of employees of the UMC Utrecht. The director of the assigned party Cochrane Nederland also services the final client (VWS) as employee of UMCU (Universitair Medisch Centrum Utrecht).

Based on the foregoing, our quality judgement is evident: this Cochrane report hardly makes a meaningful contribution to research on excess mortality.