Sunday, July 23, 2006

The Pros and Cons of Awards

The institution of the Best Paper award at STOC and FOCS is relatively recent (2002?) and it came with some controversy. Indeed, most people acknowledged that such an award (as opposed to the best student paper award, which has nearly universal support) has a series of shortcomings, such has the element of randomness in recognizing one (or a few) of the top papers in each year, and the difficulty of recognizing excellence without the benefit of hindsight. But, more fundamentally, our community likes to see itself as engaged in a group effort to advance knowledge, and it is often from unglamorous and unfashionable investigations that the next great advance comes from. A best paper award, like any award, however, inevitably rewards the individual over the group and the fashionable over the unfashionable.

All this is true, the proponents of the award (successfully) argued, but awards are useful as a medium of communication between us and other research communities. They tell people in department-wide or university-wide (or NSF-wide) committees "listen to what the theoretician guy is saying; the awards we have given him mean he is not a random guy off the street," and they tell people in ad-hoc committees "give tenure to this theory candidate, the awards we have given her mean the theory community thinks highly of her," and so on.

Oded Goldreich explains these points rather eloquently.

This month, the Notices of the AMS (the American Mathematical Society), has a discussion on a proposal to name some mathematicians Fellows of the AMS, the way other learned societies do, including the ACM. The piece includes an article in favor and an article against the proposal.

The same arguments I outlined above for/against best paper awards are given in these articles. There is, however, a noticeable addition. In arguing against the proposal, David Eisenbud says, almost in so many words, that many of those who are in favor of the proposal must be intellectually dishonest: surely they can see the many downsides of the proposal, and so their support must be motivated by their expectation of becoming fellows. This is really a new one, I don't think I ever heard such an argument when discussing awards in TCS.

It should be noted, however, that the AMS proposal involves the making of a lot of fellows. About a thousand people will be shooed in to get things started (to give you a sense, I would be eligible to be a founding fellow) and then more than a hundred fellows would be added each year. To give some proportion, ACM has about 80,000 members, and it named 34 fellows in 2005. AMS has about 30,000 members.

10 Comments:

  1. Anonymous Anonymous
    7/24/2006 02:17:00 AM

    The problem with the "best" paper is that it singles out one paper, independently of the various scenarios: there are no good papers, there is one outstanding paper, there are several equally outstanding papers. E.g., the #3 paper in FOCS may be better than the #1 paper in the next STOC (I am not sure if it even happens but suppose). What's the FOCS/STOC rule, is there one single best paper always? In Crypto/Eurocrypt, there is either one best paper or none, but the best paper award is very recently introduced so I am sure it may sometimes be awarded to several papers.

    This AMS policy sounds more like having a senior member position in some other society. Given the numbers you presented, it sounds like eventually at least 5% of all members will be fellow.

     
  2. Anonymous Anonymous
    7/24/2006 08:01:00 AM


    as opposed to the best student paper award, which has nearly universal support


    What is different with the best student paper award that it has "nearly universal support"?

     
  3. Anonymous Anonymous
    7/24/2006 09:01:00 AM

    surely they can see the many downsides of the proposal, and so their support must be motivated by their expectation of becoming fellows.

    While this is a new one, the flip side often rears its ugly head:

    Surely he can see that there are drawbacks in not highlighting the contributions of meritorious people, yet he still opposes the Fellow program. Surely it's sour grapes since he knows he is not eligible.

     
  4. Blogger Luca
    7/24/2006 09:02:00 AM

    Helger: The STOC/FOCS committee can give the best paper award to up to 3 papers, or decline to give it to any.

     
  5. Anonymous Anonymous
    7/24/2006 09:46:00 AM

    The problem with the "best" paper is that it singles out one paper, independently of the various scenarios: there are no good papers, there is one outstanding paper, there are several equally outstanding papers.

    The best paper award has minor impact on your career. Its net effect will be a small boost. It's in big awards such as Nobel Prizes, Turing awards and Field medals in which the difference between the winners and losers is huge. As Oded points out, this speaks against such awards. Feynman and Schwinger won the nober prize for QED, Freeman Dyson with an almost equal contribution was passed over. The differences in their career paths were much larger than their actual contributions.

     
  6. Anonymous Anonymous
    7/24/2006 10:44:00 AM

    Luca: ok, then it is more reasonable, although I personally would prefer to give the best paper awards with hindsight.

    Last anonymous: I am sure that best paper award have quite large career implications. As it came out in another blog far-far away, future employers count the numbers of FOCS/STOC papers of their possible future employees.

     
  7. Anonymous Anonymous
    7/24/2006 10:44:00 AM

    Luca: ok, then it is more reasonable, although I personally would prefer to give the best paper awards with hindsight.

    Last anonymous: I am sure that best paper award have quite large career implications. As it came out in another blog far-far away, future employers count the numbers of FOCS/STOC papers of their possible future employees.

     
  8. Anonymous Anonymous
    7/24/2006 11:46:00 AM

    I cannot see a single case where someone would say "we would have hired this person if it only had a best student paper award instead of just a regular paper in FOCS".

    I've had several papers invited to the special issue of a journal from a conference (the unofficial equivalent of a top dozen papers award). If you were to add or drop one, few people would notice.

     
  9. Anonymous Anonymous
    7/24/2006 02:22:00 PM

    With respect to the argument that we should have awards so as to motivate administrators, is there a good way to test this given existing data? Some things that come to mind and their problems -

    * We could survey deans as to the importance of awards. This may not catch unconscious bias, however.

    * We could look at hiring & tenure rates of closely related disciplines, one of which has awards and the other does not. How do we control for other factors, however, like the size of funding or rate of progress of the two different disciplines? (It's unlikely that such factors will be exactly the same.)

    Does anyone know someone who is versed in econometrics? These seem like the kinds of questions economists like to study. Maybe we can suggest to an economist that she may win an award for this work...

     
  10. Anonymous Anonymous
    7/24/2006 10:53:00 PM

    Best paper awards are a fixture of most major CS conferences. Theory was late to the game in setting them up. The first major award created was the Godel prize which is a retrospective award for a journal paper. It was originally instituted at least as much to induce people to revise their top conference papers for journal publication as to actually give the award (though it now is quite lucrative).

    STOC only has had a best student paper award in the last decade whereas FOCS had one for more than a decade prior to that. Our standards for student papers are different from some areas where the requirement is only that the author presenting the paper be a student. Our best student paper awards are actually somewhat less meaningful for hiring because of this (graduate students may have many strong papers that are simply not eligible).

     

Post a Comment

<< Home