What is the state of vulnerability research?

Recently, Steven Christey of Mitre asked some interesting questions. I posted a rather lengthy reply, and thought I'd share it in a more public place. The Security Curmudgeon (Jericho at Attrition dot Org) and Foofus at Foofus dot Net both had interesting replies, and (with their permission) I'm going to add them into the mix.

This is a series of open questions to people who consider themselves to be vulnerability researchers. Hopefully this will open a number of fruitful public discussions.

You would have thought that you'd be inundated with answers. I've examined the larger picture, myself, and have engaged a discussion with various groups on the subject. It seems that there is still a basic split in the picture.

1) What is the state of vulnerability research?

We should first examine what is meant by that topic. Vulnerability research has come to imply that there is an expectation of a formal (or otherwise) release of the results of such research. It seems that it is unusual for someone to experiment in the area of vulnerabilities, and yet not publish. I note that Forno's survey predicates the role of researcher as one who publishes, and I see that your questions expect the same. This leaves me at a loss. I have been involved in research of various types and kinds of vulnerabilities since 1980, or thereabouts. With extremely rare exceptions, I have never notified a vendor. When I have done so, it has been anonymously, either through such things as mixmaster, or via hushmail. I have *never* published any findings, and that will probably continue to be the case.

I have shared certain suggestions with various people, sometimes anonymously, sometimes not. If I feel that it is an item not being exploited, I may just keep it to myself. If I see signs (via what are now termed honeynets) that there are exploits in the wild, then I may make suggestions to encourage research in the area. Sometimes those suggestions are made in large groups of people (my favorite place for this is defcon), sometimes it's in private conversation. This behavior may change, now that I am not concerned about security clearances and intellectual property agreements.

Still, I imagine that I am not the only researcher thus engaged in research, where results are simply noted (for further investigation, or for remediation of local products).

There is also the question of what vulnerability research is. Do we consider every moronic cross site scripting event noted to be a result of vulnerability research? This desperate search for notoriety is amusing, but I question the usefulness of it. Certainly, for those three people that use a particular application, it may be significant, but I confess I do not think it falls into the class of research. Marcus Ranum recently posted a reply in an interesting thread on Bugtraq (entitled "Vulnerabilities in new laws on computer hacking"), which I think bears repeating. He is speaking here of pentesters, but it holds for the class in general.

"I would say that most pentesters are failed security analysts who do not understand engineering discipline and have chosen to engage in the war of band-aids instead of learning how to build correct systems. And then there are the pentesters who really are cybertrespassers at heart, who have found a financial and moral justification for doing something for money that they'd otherwise do anyhow, for free, in the wee hours of the night."

He is correct. I have been as guilty as most of amusing myself by implying that breaking into a system and leaving no trace was acceptable in the old days. It's no more acceptable than to walk through your neighbor's house in the dead of night, just because he left the window open. In the same way, finding vulnerabilities in various classes of software, or in TCP/IP, still require a standard of ethical behavior that states that we must hold ourselves above even the appearance of doing evil.

2) What have researchers accomplished so far?

It depends on what you consider an accomplishment. The full disclosure of various vulnerabilities is significant, but far too many "researchers" are using the mantle of full disclosure to rush every trifling buffer overflow into print, in order to see their name in lights. Full disclosure has done some important things. Microsoft, Oracle, Sun Microsystems, Apple, and many others are much more likely to move quickly against security and usability problems than in past. This is a direct result of the embarrassments suffered on places like Bugtraq (the Full Disclosure mailing list was not yet needed in those days). Many vendors are trying to reverse engineer security into their products. Some even have rewritten software, with mixed results. Vanishingly few are following good security engineering practices (or even good engineering practices, let alone mixing good sense about security into it). Researchers have provided the impetus for this, but they've also laid the groundwork to encourage every juvenile delinquent on the internet to believe that announcing the buffer overflow of the week, will somehow bring them fame, fortune, and a social life.

3) What are the greatest challenges that researchers face?

DMCA and its cousins threatens to throw a chill on the industry. The infusion of money into the field has also blinded many to the ethics that we should all hold to. We must all have a certain ambivalence when vendors offer bounties for new exploits, yet we know also that the criminal element does the same. Do we secure the boundaries, concerning ourselves with each and every scripting error that exposes a web page to inappropriate release of information? Do we concern ourselves with the basics of security, publishing white papers on secure programming? How do we assist our fellows in securing their applications, when examples abound in products that should be held to a higher standard?

When is it appropriate to keep quiet about a vulnerability? If a problem is exposed for which there is no readily apparent solution, what is accomplished? On the one hand, perhaps a solution is found (either by the vendor, or by other researchers). On the other hand, if it is not, then what is accomplished by exposing it?

4) What, if anything, could researchers accomplish collectively that they have not been able to accomplish as individuals?

It might be useful to have an accrediting body, like the CISSP (I know, I know, but it still has its purpose). Perhaps a code of ethics might be appropriate. For an example, please see:


It might be nice to espouse such a code. It would certainly say that you belong to a group that are concerned with a higher good, and not just the excitement of the chase.

5) Should the ultimate goal of research be to improve computer security overall?

No. The ultimate goal of research must always be pure knowledge, else it is not research. It would be nice to sign up to the feel-good notion that what we do improves the security stance of the computing world, and of the internet. Certainly, that is an effect. It simply cannot be viewed as a goal, merely as a positive by-product. In the same way, honeypots and honeynets discover anomalous attempts that may predict attacks from previously unexpected quarters, but the goal of such a device should not be simply to discover such attacks. Rather that goal should be to further the investigation of each class of attacks, an active vulnerability research, as it were.

6) What is an "elite" researcher? Who are the elite researchers?

Now there's an overused word. I prefer to believe that the questions is thus:

What constitutes a vulnerability researcher?

The answer lies in the steadfast pursuit of knowledge for its own sake. Each person brings certain skills to bear; this one has great skill with disassemblers; that one recognizes anomalies in the packet stream of a networked application. The challenge of realizing that there are weaknesses, either in a protocol, or an application, and that the light of examination and testing may expose it, satisfies in a way that little else compares to.

7) Who are the researchers who do not get as much recognition as they deserve?

Researchers who need recognition are around us by the thousands. The satisfaction of knowledge gained should be enough. There are far too many "researchers" shouting at the top of their lungs in the marketplace. Any benefit to doing so has long passed.

P.S. If you're further interested in letting your voice be heard, check out Richard Forno's disclosure survey at http://www.infowarrior.org/survey.html

Been there. I recommend it to all who are in the field, and have not yet participated.

Etaoin Shrdlu
Last modified: Thu Mar 2 07:17:17 PST 2006