The state of vulnerability research, continued...

Brian Martin (aka The Security Curmudgeon, Jericho at Attrition dot Org) answered a poster who wondered at Steven Christey's impartiality. Brian has also posted about this separately, at his blog on the Open Source Vulnerability Database.

On most days I would agree with you [the original poster] , and come up with a reply along these lines =) However, since I am a cynic AND involved in the same 'scene' and core interests Christey is, I think I understand this fairly well.

Why do I think Mitre should be coming out with answers, not questions?

Vulnerability research is straight forward. There isn't a lot of black magic and secret arts when it comes to finding vulnerabilities. For the most part, 99% of vulnerabilites are very well documented (even if the 'researcher' doesn't document it), easy to understand by others in the field, and leave little to imagination. It has been years since we've seen a truly new class of vulnerability surface. If I post details of an overflow of *any kind* to this list, there are a hundred folks that can digest what I post in seconds, then go to town on me for not going into details, not looking at VectorX, FunctionY or Z.c =)

The other side of vulnerability disclosure is the human element. The sociology and mindset behind what we do, and why we do it. This is the angle that has interested me for years, and the type of book I will grab before any 'technical' (generous term usually) security/hacking book. Not only are there dozens of questions that can be asked of the researcher about his mindset and ethical views, there are countless other people involved in the process. Does the researcher have partners? Is he an employee of a security company? What vendor is he dealing with? Which vendor is it? How many people is he dealing with on the vendor side?

All of that will factor in to how a disclosure plays out, along with a thousand other variables. Trying to understand that is not easy (if at all possible). If Christey or MITRE said "we understand the state of vuln research, here it is", you and I would both be flaming him until we pass out at the keyboard. The fact that he *knows* he doesn't know is actually an impressive quality. The fact he will actually *admit* that he doesn't know, is staggering. The fact he will ask the community their thoughts (hey, peer review?!) and factor it into his own research and answers.. wow! If you think Mitre should be coming out with answers, without asking these questions, then I think you should step back and think about the state of vulnerability research.

I don't say this as an insult or a derogatory comment at all. A couple years ago I would have posted the same thing, without realizing just how complex of an issue this really is. Since I started working with OSVDB.org and running such a database, my appreciation and understanding of the complexity of vulnerability research and disclosure has skyrocketed. That skyrocket has now put me somewhere between "i don't have a fuckin clue" and "i think i have an inkling of understanding here". I say this because behind the scenes, after you read the bugtraq or full-disclosure post about an issue, the real research frequently takes place. If you or anyone else knew just how many mistakes, oversights and shortcomings occured in the average disclosure, you may be surprised. If Christey or I had a nickel for everytime we mailed the other about a flaw or oversight in the other's database, we'd both be surrounded by a dozen hookers feeding us tropical fruits and exotic drugs on our own private island.

The current state of vulnerability research SUCKS ASS. Yes, quote me on that. I say that in the context of the overall research we see. I say that based on a cynical and jaded perspective I have after spending countless hours clarifying, correcting and further researching issues that have already been 'researched'. I say this after dealing with dozens of vendors who tell me that *I* am wrong because OSVDB has information culled from other sources, and that say *I* am a moron for thinking X is vulnerable to Y and Z is the impact. Doesn't matter that half the time (or more) the vendor is wrong and sending me a snap judgement mail defending their product. Doesn't matter that sometimes they are right, but in figuring that out I find several other vulnerabilities in their product. Doesn't matter that CVE, OSVDB, SecurityTracker and others spend a few days to finally figure out and resolve discrepancies in the most boring and simples of disclosures days before. Doesn't matter that one such vendor had an office 1.4 miles from me and the only thing that stopped me from pissing on their front door was 18 degree weather.

Think about it.. why would ANY 'vulnerability report' require that much followup and analysis? One of the many things on my 'to blog about' list is a subset of Christey's question. Why do people think they are actually doing 'vulnerability research' by pasting a ' into a field and getting an SQL error? Hello.. that doesn't mean you can inject SQL queries! Pasting the standard