Throughout history, for most of the time, world population has been growing, with the rate of that growth also increasing along with the transitions between the foraging era, the farming era, and the industrial era. While declines in the population of specific regions or countries have been more commonplace, declines in the overall world population have been quite rare and haven't lasted very long. The two main causes of such occasional reversals in the overall trend of growth have been pandemics and wars, in that order. While both the data about historical world population and the death tolls of various pandemics and wars are uncertain, what we do know is sufficient to at least give us a sense of the order of magnitude of the worst population crashes in history.
Shifting focus from the past to the future, many of the concerns about existential risk have a natural weaker counterpart in the form of concerns about population crashes which are not large enough to tip the species over the precipice of extinction. Since humans have never actually gone extinct or come anywhere close to it during or after the agricultural era, the only base rate information we have comes from the history of global population declines. It's therefore worth looking over the biggest cases to understand their causes along with their magnitudes.