Unpacking National Crime Statistics

More frequent crime reporting would reduce the confusion that is clouding the presidential election

Sep 18, 2024

Busy couple of weeks here at External Processing (please check out my latest essay on measuring war crimes, published last week). A few days ago, the Bureau of Justice Statistics (BJS) released the 2023 National Crime Victimization Survey (NCVS). The NCVS is one of the two key sources of national crime statistics, the other being Crime in the United States which is official police data reported by the FBI that should be released shortly for 2023.

Now, anytime you have two different sources of information on one complex topic, there are bound to be some inconsistencies in the state of things. This is true for economic data on unemployment (more on that below) so we should not be surprised it is sometimes true for crime statistics. Jobs data is released monthly, and any inconsistencies are mainly smoothed out. Crime data is released once a year, and as a result, inconsistencies can be mistaken as more problematic than they are.

So, one big point of this essay is to show why the US needs monthly crime data and I am going to go through the recent NCVS releases to make the case that if the resources were there for monthly data collection, the things we are wringing our hands about today would generate a lot less handwringing. In doing so, I want to also make the case that the differences between the NCVS and UCR data are getting blown way out of proportion. Crime in general, and violence in particular, has declined substantially.

NCVS Data

Just to make sure we are all on the same page, last Thursday, September 12, the Bureau of Justice Statistics released the annual National Crime Victimization (NCVS) data for 2023. As the name implies, the NCVS is a huge survey of households (150,000) and individuals (240,000) about their recent personal experiences with crime. This week, we expect the FBI to release the official national crime statistics from the Uniform Crime Report (UCR). The UCR documents crimes reported to the 18,000 local law enforcement agencies in the US.

In summary, we have two sources of national crime statistics collected by two different methods, both released about nine months into the year.

Now, before you say, that’s a crazy way to figure out how much crime there is in the United States, let me say, it’s really not! In fact, it is very familiar to you. This is very similar to how jobs are counted in America.

Noah Smith, per usual, has a clear description of how we count jobs. A couple of days after the end of each month we get what everyone calls The Jobs Report but that is officially The Employment Situation which isn’t exactly a gripping report title. But whatever the name, the report describes the prior month’s unemployment rate, and how many Americans were looking for a job, how many found a new job, etc.

My guess is that given the way this is reported, like commandments trotted down from the mountain, most people assume that we have excellent administrative data which allows a count of exactly how many people have jobs. It’s actually a mix of survey data and administrative data, but mainly survey data.

Counting jobs each month is a three-step process, costs some money, but is valid and generally reliable.

The U.S. government Bureau of Labor Statistics has three basic methods of determining how many people have jobs:

It calls up companies and asks “How many workers do you employ?”, and then tries to extrapolate from that. (This is called the Current Employment Statistics survey, or the “payroll survey”.)
It waits for states to report their unemployment insurance claims, and then tries to extrapolate from those. (This is where the revisions to the CES come from.) Occasionally, those revisions can be quite large.
It calls up households and says “Hey, do you have a job right now?”. (This is called the Current Population Survey, or the “household survey”.)

My main point here is not subtle. These are estimates of the jobs market, and as with any estimates, there’s variation. The more often the data are collected, the less effect any particular month has, and the clearer the trend. If we want a clearer understanding of crime, we should do the same thing.

There is also a subtle point. That is, there is an inherent riskiness in only reporting national statistics once a year, as we do with crime. Once in a while—due to the vagaries of the universe—the data will present an extremely unlikely scenario. A 95% confidence interval means that all else being equal, one time out of twenty you can expect a result outside the confidence interval. If that happens with a monthly series, the trend will almost certainly self-correct. If that happens with an annual series, it can create chaos.

Counting crime the same way we count jobs is entirely feasible.

This, of course, is not how we count crime. There is one survey, conducted over a year, and reported annually. And one administrative data source, that reports annually.

And there are insufficient resources for even this very modest effort to collect national crime statistics. The FBI is asked to cajole 19,000 local police agencies into voluntarily reporting data that is becoming increasingly complex over time. Now you might argue, and I would definitely argue, that collecting data on crimes is not only the foundation of effective policing but the only way to do effective policing, and thus local police should be doing this anyway.

But the reality is that 40% of law enforcement agencies have 10 officers or fewer and these reports are a big lift. (Can you imagine Sherriff Longmire and his team of three reporting NIBRS data? (But hey, Sheriff Longmire, look at how much data you quote in this clip, you are already data-driven, all you have to do is write it down). Also, anytime I mention sheriff's I feel an obligation to link you to Jessica Pishko’s amazing Substack and strongly encourage you to buy (hardback and full price) her new book on sheriff’s The Highest Law in the Land which shipped Monday, September 17. Or go see the launch in person in NYC.)

But I digress. the point is that *a lot* of additional resources are needed to improve local police data reporting. The good news is that once that process is improved, it is a much smaller lift to turn the annual series into a monthly series.

That’s the administrative data side. On the survey side, BJS has curated an exceptional survey (the NCVS) that with sufficient investment could be turned into a monthly crime victimization survey. The US has best-in-the-world statistical agencies, and when it comes to crime, these statistical agencies are limited only by funding. The US could, with great precision, determine how much crime there is, in near-real time, without increasing surveillance and threatening democracy. We could just call people and ask them about their experiences.

The 2023 NCVS

This brings us to last Thursday’s Bureau of Justice Statistics release of the 2023 National Crime Victimization Survey. The key to understanding these results is to understand that this is a report about victimization in both 2022 and 2023. I’ll unpack that statement below and in doing so, make a case for monthly surveys. Here are the results that the authors spotlight.

The headlines are:

A small decline in the overall rate of violent victimizations (a decline from 23.5 victimizations per 1,000 persons age 12 or older in 2022 to a rate of 22.5 in 2023), about -4.3%.
A small increase in the overall rate of property victimizations (an increase from 101.9 victimizations per 1,000 persons age 12 or older in 2022 to a rate of 102.2 in 2023, which is essentially zero change.

So, as a top line finding, there was little change in crime from 2022 to 2023.

But underneath that, there is a lot of encouraging news.

Violence crime excluding simple assault is down 11.2%, including domestic violence (-16.3%), intimate partner violence (-35.3%), violent crime with injury (-8%), and violent crime with a weapon (-17.2%). The notable increase is in stranger violence (+7.5%).
NCVS asks respondents about crimes reported to police which tend to be more serious than other types of victimizations.1 I calculate a 27.4% decline in violent victimizations (excluding simple assault) and a 34.8% decline in robberies reported to the police.

How does the NCVS Compare to the UCR?

The 2023 Crime in the United States is not out yet, but we know from prior preliminary releases approximately what is in the report.

Violence crime all of types was down 5.7% in the UCR, compared to a 4.2% decline in the NCVS.
Property crimes were down 4.2% in the UCR, compared to 0% in the NCVS.

All of this is very encouraging! But a look at recent trends suggests this kind of consistency is not necessarily the norm.

Comparing NCVS data with FBI data is complex is a dicey business. Still, the reality is that the two trends are being compared in their least nuanced form in the ongoing presidential debate. One side describes a hellscape of spiraling violence (relying on the 2022 NCVS), and the other side describes a sunny vision of a COVID-19 crime wave that has completely receded (relying on preliminary FBI UCR data).

So let’s look at the trend (and I heartily acknowledge that there are a huge number of caveats to the graph below).

What I’ve done here is to truncate the bottom half of these graphs, so each is on a visually equivalent level. The picture suggests that NCVS is much more volatile over time than the UCR. For instance, from 2009 to 2010, the NCVS showed a 15+% decline in reported violent victimizations while the UCR showed about a 5% decline.

But two years stand out.

In 2022, the NCVS showed a 46% increase in violent victimizations while the UCR showed a 1.7% decline.
In 2020, the NCVS showed a 22.8% decline in violent victimizations while the UCR showed a 5.6% increase.

But here’s the rest of the story, upfront. I actually don’t think these differences are entirely inconsistent with the facts on the ground. But I do think when you work your way through the logic, it becomes apparent that if these data were released monthly, much of the inconsistency between the two series would become transparent, and less controversial.

How is it possible that two data series reporting wildly different results from the same time period are not inconsistent? Read on!

Unpacking when the NCVS and UCR data diverge

First, a quick brief on the way the NCVS is administered.

The NCVS is administered throughout each calendar year, with similar numbers of interviews each month;
The survey asks respondents about their experiences over the last six months, beginning the month before the month of the interview (most interviews are conducted within the first two weeks of each month);
People interviewed in the first half of the year will report crimes in the year of the survey and the previous year (interviews in January, for example, will ask about July through December of the prior year);
A declining percentage of respondents will be asked about the last six months of each year.

Each survey then covers an 18 month period, rather than a 12 month period. Jeff Asher writes very clearly about what he calls the NCVS vs UCR conundrum and created this extremely useful chart showing what period each of the NCVS reports covered.

Now, the two key years to focus on are 2019 to 2020 and 2021 to 2022. These were years of rapid increases and decreases, and the long time horizon tends to obscure inflection points where crime increased and decreased rapidy.

Now, I know y’all love it when we do some math here, so let’s go! One way to think about the NCVS data reporting calendar is to think about it as person-months instead of people. What that means is each person is asked about crime in six different months.

Above, I said there were 150,000 household interviews in each survey. To make the math simpler, let’s say NCVS does 12,000 interviews each month, which gets you to 144k total interviews over a 12-month year. Again, note that each interview is about the preceding six months, excluding the month the interview is administered. So that means that the distribution of interviews each month looks like this:

Source: Author’s calculations. See 2019 National Crime Victimization Survey: Interviewing Manual, https://bjs.ojp.gov/document/manual2019.pdf

What this says is that the January 2021 surveys are administered entirely in 2020, from July through December. The February 2021 surveys ask mainly about 2020, but not about July 2020—the January 2021 interviews are the only interviews about July 2020 so there are few of them. The March 2021 interviews ask about September 2020 through February 2021. No new interviews are added for July or August 2020. You get the idea.

It’s important to note that the same logic applies at the end of the survey. The December 2021 surveys, for example, are the only surveys that ask about November 2021. And, there are no interviews in December 2021! (Interviews exclude events in the month of the interview). Got all that?

Together, all of that gets you a filled-in chart that looks like this:

Source: Author’s calculations. JohnKRoman.Substack.com

The top row is the month and the year that the interview refers to, the 18 months from July 2020 to December 2021. Looking at it this way, there are a couple of interesting features of the 2021 NCVS used in this example.

A huge number of interviews in the 2021 study were undertaken in 2020: 252,000 interviews refer to 2020 months out of 864,000, or about 29%;
There were very few interviews in the final six months of 2021. Because the survey asks about the prior month, there are no interviews in December 2021.
There are fewer interviews in the last six months of 2021 than there were in the last six months of 2020, only 180,000 interviews (20.8%).

None of this is news to BJS of course, which explicitly states, “Therefore, the 2021 survey covers crimes experienced from July 1, 2020, to November 30, 2021, and March 15, 2021, is the middle of the reference period.”

Not to dwell on this point, but this is a critical point. When a survey is presented to the public that is labeled as the 2021 survey, it is fair to assume that the midpoint is around June 30th, not more than 100 days earlier, on March 15th.

The 2022 NCVS

Now, let’s turn to the survey that is causing so much consternation today, the 2022 NCVS which showed a 46% increase in violent victimizations. Republicans are citing the massive increase in the 2022 NCVS as evidence of escalating crime and Democrats are leaning into the 2022/23/24 UCR data showing declines.

One point to make is that by all indications, violent crime peaked in 2020. Violence declined slightly in 2021 and again in 2022. Because of changes to the UCR in January 2021, we don’t know much about what happened in 2021. It is certainly possible that violence in 2020 continued at the elevated levels of 2020 for a while and then declined in the second half of the year. If it was the case that much of the 2021 decline occurred in the second half of the year, then the 2021 NCVS would show an elevated level of crime—oversampling the high rates in 2020 and the first half of 2021 as described above. The 2022 NCVS then would capture the lower levels in the second half of 2021 and all of the 2022 decline.

Another point is that the NCVS is historically more volatile than the UCR, particularly in periods of rapid increase or decline, as the figure above shows. There is a logic to these discrepancies. A little math shows why this may matter a lot.

We know that in a typical year, about 40% of violent crimes in the NCVS were reported to the FBI, and 30% of property crimes were reported. To make this more straightforward, let’s assume that 50% of crimes are reported to the police, and 100% of crimes are reported to the NCVS.

Now, suppose for some baseline reporting year, the UCR reported 2 crimes and the NCVS also reported 2 crimes. Now, let’s assume that the following year, crime went up by 50% in the UCR. This would result in 1 additional reported crime, increasing the total UCR crimes to 3. However, in the NCVS, there would be one additional crime (the one captured in the UCR) and an additional crime that is only reported to the NCVS. So, when crime goes up by 50% to 3 in the UCR, it goes to 4 in the NCVS.

Given the limited reporting of crime to officials, the NCVS and the UCR should not be expected to show the same percentage change when crime is volatile. This is a stylized example, but you get the idea. In times of real movement in crime, up or down, the NCVS should show bigger changes.

Putting these two explanations together, it’s not hard to imagine a scenario where NCVS effectively overweight a period where violence is rising rapidly (July-December 2020), and underweights a period where violence is rapidly declining (July-December 2022) and thus you have two measures of the same period measuring different results. Because they aren’t actually measuring the same period!

I think it is clear that a monthly series would solve many of these problems. If there was The Crime Report similar to The Jobs Report, then an equivalent number of surveys would be conducted every month and the vagaries of the reporting periods that currently confuse readers would disappear. The alternative is the world we currently live in, where a hugely consequential presidential election is being driven, in part, by dueling narratives about national crime statistics, because of the serious complexity in trying to align the two official national statistics.

What Does it All Mean?

Well, gentle reader, I think it means that the tenor of the presidential campaign will unfortunately not change after this data release. If you think America has gone to hell, there’s data to support the claim it stayed that way in 2023. There is far more evidence, in my opinion, that public safety has dramatically improved. But I expect the war of words to continue.

There is a lot more to say. Stay tuned.

External Processing