Comments on Jeff's Search Engine Caffè: Ten Myths of Computer Science Research

Interesting thoughts, all around.

I tend to...

2008-10-02T17:31:00.000-04:00

Interesting thoughts, all around.

I tend to agree with the "your code won't be used in 5 years" statement. My feeling is that this has a lot to do with the "science" aspect of computer science, rather than the engineering aspect.

With engineering, you are trying to design systems for stability and longevity. You are not really trying to make things extensible or flexible. Except for maybe some sort of plug-in architecture. But for the most part, user requirements do not really change. And so it makes sense that code will still be used 5 years later. Modulo those plugins, the user is essentially doing the same thing with the software 5 years later.

With science, on the other hand, you are continually testing, discarding, reformulating, retesting and synthesizing new hypotheses. In a sense, very little code is written to ever run twice, because once you've tested a hypotheses, you move on to the next one. Why would you keep running the same experiments, over and over? Maybe if new data were continually made available? But even then, you might run your code for 2 years, or 3 years. After your 5th year of running that code, how many papers or posters or journal articles are you going to get, saying "yup, the hypothesis continues to bear out"?

I think that's all that they're saying, here.

Perhaps the exception to this rule is for lower-level, data structure or generic algorithm software development. For example, if you've spent a fair amount of time writing data structure libraries (b-trees, hashsets, etc.) in C, you probably will be reusing that code not just in 5 years, but in 10 years. Similarly, if you've implemented some low level machine learning algorithms (Adaboost or whatever), you will probably reuse that as well.

But those become libraries, into which you plug your main code, your hypothesis-testing code. And that main code is the code that you won't be running in 5 years. And that main, hypothesis code is where all the interesting stuff is happening, where all the "real" development is happening. Not in the b-tree library for C/C++.

"Science" code is a very different creature from enterprise, production code.

Bob,

I asked Dave Jensen about the quote an...

2008-10-02T10:02:00.000-04:00

Bob,

I asked Dave Jensen about the quote and it is from Paul Cohen.

I don't think Paul was saying that there isn't value in well designed and documented software, especially in the commercial sector.

His message is that in the long-run, whether it's five years or fifty, that the code won't survive. However, the ideas in it and the findings produced by running it will endure. This is especially true for people conducting experiments in academia where ideas and not software is the product. In this context, software just a tool to produce empirical results and prove theories.

The Sciences of the Artificial was mentioned to refute the claim that was made that Comp. Sci. isn't science because we don't study 'real' subjects.

The Hamming talk is one of my favorites, thank you for pointing it out.

From the quote, which I don't see in their slides,...

2008-09-30T16:51:00.000-04:00

From the quote, which I don't see in their slides, it sounds like these guys (or whomever they're quoting) have no clue about commercial software. Or even reasonably well-written and documented academic software. I wrote academic software in 1992 that's still fairly widely used today. Many of the classes in our current production software were written more than five years ago.

Real software in real products tends to be incremental rather than being continually rewritten from scratch by each new grad student that can't figure out what the previous one did.

Sergio: the myth was that CS isn't science. I believe the authors by calling it a myth are implying that it is science.

The Gretsky quote is what my old VP at Bell Labs used to call the "pee-wee soccer effect" of everyone following the same ideas.

Simon's Sciences of the Artificial is one of my all-time faves, but I don't see how it's relevant to this discussion.

While I'm not saying Kuhn's book on scientific revolutions is wrong, it's used in practice to justify nonsense by authors who think they're revolutionaries.

PS: A good practially-oriented and fun read on this topic is Hamming's advice to young scientists.

Nothing like replying to myself after rereading th...

2008-09-30T16:42:00.000-04:00

Nothing like replying to myself after rereading the text. Obviously I interpreted the first myth backwards. Please ignore. :)

I agree with most myths mentioned. However I reall...

2008-09-30T11:34:00.000-04:00

I agree with most myths mentioned. However I really can't understand the first one - "CS isn't science".

Sure there is CS research that is badly designed thus hardly can be called science. This happens in all fields.

Can you give a little more detail of the reasoning behind this assertion? Thanks!