A recent post (on spotting bugs in papers) at Claire Mathieu's blog reminded me of something a colleague told me recently: he noted that conferences are requiring more mathematical rigour from submitted papers, than they used to (say, about 10 years ago). Where previously we could get away with “proof (sketch)” there is now a stronger expectation that the whole thing will be written down, either in an appendix or on Arxiv. If we make a mistake we now face rejection; no longer do we just try to convey the general idea, but it’s required that the details be done correctly. My own limited experience agrees with the theory that this is a trend, but I am mainly just passing on hearsay. (Let me know if you think this is indeed a trend.)
This all seems fine, and yet... I have concerns. The main one is the risk that conferences threaten to usurp the traditional role of journals, in purporting to showcase papers that are guaranteed to be correct. The traditional life cycle of a paper envisages a conference version that publicizes the idea, which results in the author(s) receiving feedback on the paper, that may inform or contribute to a subsequent journal version, whose main purpose is to be correct, in some cases setting the record straight if there are bugs in the conference version. If we’re going to insist on correct conference papers, it raises the question of whether the guarantee is as strong as it is for journal versions. If so, there is no point in the journal paper.
Should we mind if a few conference papers are buggy? I’m not sure that’s such a big problem. If the paper attracts interest it will receive sufficient scrutiny to detect errors, and furthermore will be submitted for journal publication (as warranted by the interest it attracts). Thus, a buggy paper will either get exposed, or else no-one will read it. Moreover, is it even desirable that our literature should be entirely pure — is there not room for the human factor to obtrude? There is mild entertainment to be obtained from the occasional reminder that a paper may have been driven by over-enthusiasm on the part of its authors, and it can be reassuring to see that even our more esteemed colleagues sometimes get things wrong.
Subscribe to:
Post Comments (Atom)
10 comments:
I think a problem with this view that I have seen pointed out before is that it is hard to get credit for proving something that someone already claimed to prove. Thus if a paper is buggy (but the general idea correct), it also acts as a deterent for any one else to actually do the work (either getting no credit, or having to point out loudly that the previous work was flawed/incomplete so that they do). The authors themselves may not want to draw attention to the bug, or may not see it as such.
This problem only exists in so far as our community runs on "credit", but I do this this is currently largely the case.
Do you realize that if we are allowed to publish incomplete/buggy conference papers, then it is quite likely that people will be hired into their first jobs on (what the previous commenter referred to as) "credit"?
Already this happens. Someone has a couple stoc/focs papers, none of which are complete, but looks much better on paper than someone else without such papers but who only has complete results. These two cases may not really be comparable, but the former will look better in today's hiring standards.
Well, pardon my lack of experience, but I infer from this post that I did not understand the role of conferences at all thus far.
I've always been told that in TCS, conference papers were more important than journal ones (although I never really got why). This is partly the reason why I submitted to conferences -- and also, for the joy of travelling and meeting other people in the field -- but I always took the same care in writing those papers as I did in writing my journal papers. Had I known that those conference papers were not to be taken that seriously, I would have been less careful I guess; or would have a longer track record; or would have submitted to a journal directly.
Unfortunately, conferences are expensive, which is one of the reasons that led me to submit to journals instead. What's more, I've always found it dishonest, and unfair to say the least, that two barely different versions of the same paper would count as two separate publications on a resume. I thought you were supposed to publish a journal version if you had at least a few new results, or if you did not have enough space to write detailed proofs. I have a conference paper that I never bothered to turn into a journal version because I had nothing more to add and the conference version was sufficiently thorough. But after reading your post, it seems that it was bad strategy on my behalf; should I submit it again to a journal, just because everyone else seems to do so?
Let me try to answer Anon 3's questions - for most CS journals it seems that there is no requirement to add content to what was in the conference version (in contrast with Economics for example where you are usually required to have extra content). And the standard argument for making a journal version is that the journal publication is certified correct (maybe you think the conference version was fine, and the reviewers thought so too, but a more careful journal review might still turn something up that needs to be fixed; it's happened to me, anyway...)
The research community mainly gives you credit for conference papers, but universities are not the same thing, and often look for journal papers. But the extent of that preference depends on which university you are at (or would like to join).
The factors in hiring are very far from that of paper counting in either conferences or journals. Paper counting is a common misperception - it is quality, not quantity, that is the biggest factor. That said, conference papers do play a vastly more important role in hiring of junior faculty than journal papers, simply because of their relatively short determinate time to publication. The time between submission and when journal papers appear has a long tail and one cannot expect a young researcher to have much that is far along in the journal pipeline. That said, it is always a good sign of a candidate's thoroughness if they already have some work in journals.
These days, especially with things like the CRA statement on the importance of conference publications in promotions, university tenure and promotion committees do give considerable weight to papers in top conferences. However, they also will look for a good percentage of those conference papers to turn into journal papers in good journals. This is a longer term confirmation of the thoroughness and depth of the author's work (particularly since conference papers have strict page limits). There is a pretty well-defined pecking order of journals, too, which adds extra information for a committee.
I agree with Paul. The ethos for conference submission has changed over the years. When I was first on a FOCS PC in the early 1990's, papers were still submitted via multiple hard copies sent to the conference and then re-sent to PC members after the deadline. The 8 week turn-around meant that there was little opportunity for sub-reviewing - except for sub-reviewers at one's home institution. The PC+sub-reviewers simply did not have the time to check so many details as they do now. Still, even then, the PC was uncomfortable with the lack of detail in some of the submissions and rejected at least one of the papers that was initially highly rated and by well known authors, because some of the claims came into question. Given that we now have greater ease of using sub-reviewers to check details, and fast and easy dissemination of full versions of submitted papers, there is no reason for a PC to make guesses about half-written work. That dissemination also allows a certain "crowd-sourcing" of the checking of details that is outside the PC process.
I think that the extra details do improve the quality of PC decisions but others may quibble. I am pretty sure that in the past there were good papers that were half-baked at submission time that only became solid between the submission and the camera-ready copy (another out-dated concept).
Do I trust conference papers more because of it? A little but not nearly as much as journal papers. Not every issue can be thoroughly vetted in the short time-frame of conference reviewing.
Paul, those are good points, thanks for adding them!
I diasagree with Paul G's comment that in CS, there is no requirement to add content in going from conference to journal form.
Certainly this is not true for ACM. See http://www.acm.org/publications/policies/sim_submissions, which asks that "the paper has been substantially revised (this generally means that at least 25% of the paper is material not previously published; however, this is a somewhat subjective requirement that is left up to each publication to interpret)"
@Paul Beame: Could you specify what the "pretty well-defined pecking order of journals" is? Maybe apart from JACM and SICOMP, what journals are considered 1st tier, 2nd tier, 3rd tier?
I am honestly interested in this. I asked around a little bit and the responses I got are much more diverse and fuzzy than the responses about conferences.
Most of the fields (including Math) do not even count conference "extended abstracts" as real publications. So I wonder why we (CS) are fighting so hardly for this "half cooked results" madness? In this respect, CS is an exception. But is it a good exception? The argument of Paul Beame does not convince me: there are journals in TCS that handle papers in 3-4 months, and not only the 3-rd hand journals. Also, young people in Math and other fields somehow imagine to be haired, without conference publications. The "illness" seems to be in our (i.e. of hiring committees) view at things, not in a real need.
From a mathematician's perspective it looks a bit different. Conference papers are usually refereed and published in a regular issue of a journal. In a sense they are no different from any other mathematical papers. However, when it comes to research evaluations such as the forthcoming British REF, they are tarred with the same brush as the kind of papers that Paul is seeing a trend away from. This has had the effect of discouraging mathematicians from describing important work at conferences, or from publishing it in the conference proceedings.
The trend in mathematics conference publishing is to downplay as much as possible the fact that an issue of a journal is a conference proceedings, even going so far as to allow submissions from anybody (whether at the conference or not) on the conference topic.
So we may be standing on your sidelines but are probably cheering for the trend.
Post a Comment