Skip to main content

selection-bias-and-information-security

I read an interesting post about risk strategies and selection bias that made me think about some short term thinking often seen when investments in information security are deferred. Patri Friedman discusses poker strategies in light of selection bias:

You see that if you look at the performance of many businesses w.r.t. a risky practice that is a bad gamble, you can find the slightly negative trend line. But what happens if you consider only those businesses still around? This happens accidentally all the time - after all, its much easier to survey those businesses. The result is that you eliminate the worst failures of the practice you are examining, leaving a falsely positive impression.

The same thing happens in the poker tournament world. Certain styles of play trade EV for variance, allowing people to build up huge stacks occasionally, but usually go bust. Such players often win tournaments - but that doesn’t mean they are playing right. How many times do they fail for each victory? Do they fail more often compared to the money they win than a more conservative player? Some of these “maniacs” are smart players, carefully choosing their gambles and maximizing their returns. But some of them, frankly, are just maniacs, gambling and getting lucky, and giving the false impression that high-variance play is the way to go, because we don’t notice the hundreds of people playing that way and losing.

Since it is unlikely that one executive would say to another: "We just got hacked and lost all the HR information on our employees because my password was "Britney" and someone guessed it", there surely is some selection bias occuring in information security.

I wonder if laws that require disclosure will remove this selection bias. Since TimeWarner bought Turner here in Atlanta and I know a number of people that worked there, I probably now know someone who got a letter saying their personal information was potentially compromised. Chances are that some high-level execs got those letters and it will be fodder for discussion on the golf course or cocktail party.

Here's more:

Nor is this limited to poker or business. Consider an activity that usually goes fine, but has a small chance of resulting in a mysterious death. (Perhaps some backpacking or rock climbing practice). How are we ever going to hear about them? The only people who describe the practice are those who survived. In general, anytime people will tell you about success but not failure (or vice-versa), you are going to get a distorted view. This is why double-blind placebo controlled studies, or carefully tracking all data, are so important for getting at the truth.

Current rating: 1

Recent Posts

Archive

2024
2022
2021
2019
2018
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008

Categories

Tags

Authors

Feeds

RSS / Atom