How to Read a Computer Science Paper?

Many students a.k.a. research newbies just like me may come up with such question from time to time throughout their research process. Every time when they try to grind themselves inside and to catch up with as much as they can understand, they easily feel frustrated and exhausted, hence sometimes they may regard research as a grinding process with little fun, and quit for a job position, or to pursue a MS degree–I have to admit, such process is reasonable and research is bound to be some kind of hard games of the few: given that you do not quit when your project is half-done, otherwise your advisor may get really sad :(

I have heard our fellow classmates’ questions about how to read papers, and teachers’ complaints of students’ slow rate of paper reading, and a common lack of personal research insights. The teachers usually attribute such students’ failure, i.e., failure to read papers quickly and to catch the main idea, henceforth expressing their personal feelings and research sense, to the lack of trainings on critical thinking. I partly agree with that. But, alas, I would like to attribute their failure more to failure on understanding how a nice paper is written, or how a few pieces of rubbish and bullshits are churned out. By enhancing such knowledge, I believe they might be more adequate and capable in paper reading.

Therefore I will show you how a standard form of paper that is familiar to me is arranged, and how to read a paper, well, more importantly, how to find the drawbacks the authors are trying their best to hide, and to challenge and judge the paper.

Take papers in security or software as an example, various may it be in arrangements, they usually have some common sections: Abstract, Introduction, Related work(background), System design, System implementations, Evaluation, Conclusions, Bibliography…

If you read the paper as it is arranged, congratulations, you might never find any vulnerabilities in their research–a good research paper are usually well-arranged, and no one would like to expose their drawbacks too explicitly. Only when you break their arrangement should you find something they are trying to hide.

I will introduce these sections in the order I prefer when I am reading a paper. I still remember one of the most horrifying moment when I started reading a paper 3 hours before the reading group start, and I read, analyzed, and made slides, and spent 45 minutes introducing the paper. I have to say, THIS IS STRONGLY NOT RECOMMENDED, and PREPARE WELL BEFORE PRESENTATION. The classmates, advisor and the authors will show gratitude to you, I promise.

Claim: this paper is intended for new students trying to do research with adequate English reading skills(if not, please practice). This article only represents my personal opinion as a student with only 1 year of research experience, and only serves as a reference, not a tutorial.

Abstract & Introduction

Abstract and introduction might be the very first noun a mentor may mention on his/her first course on paper reading.

I used to regard abstract as the introduction to the introduction, after one year’s training, however, I would like to regard the abstract as an introduction to the paper itself. Yes, you didn’t get me wrong. The introduction to the entire paper is Abstract section, not the Introduction section. Put your hands down and we will discuss this later. When submitting a paper, abstract is often submitted before the full paper is submitted: the submission has two deadlines: abstract ddl and full paper ddl. The abstract usually introduces:

  • how did the idea come from, they may discusses the current situation of the research, and then say: “therefore,…”
  • what kind of thing did they present to us: e.g. “we present frostwing98, a blog writing machine which utilizes human intelligence”
  • what kind of contribution did they claim. This part is usually twofold. They first introduce the techniques integrated in their system, then introduce impacts of their systems.

This is pretty much the short abstract, which covers the main components of a research: motivation, challenge, contribution(including novelty), and little details about the system itself. These components are also good metrics for evaluating a good paper.

Then the introduction part. I am here to answer the previous question. The Introduction section is not a introduction of the paper. It is an expansion to some part covered by the abstract, i.e., the motivation, challenge, and the contribution. Introduction section usually discusses the current situation of researches in detail, then the techniques the paper used, and make judgments on previous works. But these parts are not so important. You have to pay attention to the last two paragraphs, starting with a list of their claimed contributions.

You may have to keep the contributions that they claimed in mind. When reading the following part, you have to think well whether their contribution still holds true. Some sneaky writers may change concepts, or alter the original problem, making their contributions more of a bound-to-be rather than intellectual results overcoming challenges. Then you may think whether this is because the problem encountered is inevitable and unsolvable. If problem is inevitable and unsolvable, we may concede their drawback but still regard them as good paper. If not… well…

When you see this, you may have some certain feelings whether this research is good or not.

Then, let’s find the related work or the background part. This part may be in the first two pages, or the last one.

In this section, there are two types of related works.

One is the kind that author may agree with or depend on. e.g. Some techniques the authors adopted, or some similar work done by other teams. But when the authors mention the similar works with a neutral of positive tone, they usually claim their difference from the similar works, and may claim that their are better in the problem defined in their own papers.

The other one is the most interesting part in a paper, from my perspective. The author lists several works and criticizes each of them. Usually the judgments include words like slower, lager overhead, more vulnerabilities, lack of scalability…… The most interesting things is that there is a chance that you find out the paper itself cannot solve such problems either. Well…… Okay. Sarcasm?

Design

I usually recommend to take a quick look at this part, especially the figure of system design and explanations. Then jump to evaluation and conclusion.

After that, you may see the detailed implementations and design.

Evaluation

This is indeed a vital part. A good research, from my opinion, should have good definition of novel problem, solid solution, and good and thorough evaluation. A nasty evaluation ruins a good work, in that it not merely upsets the viewer, but also fails to show how good the work is. To see how a good evaluation is like, I would recommend you to read:”Morpheus: A Vulnerability-Tolerant Secure Architecture Based on Ensembles of Moving Target Defenses with Churn”(ASPLOS’19), if you have some system or security background. This paper was highlighted by me and I gave quite positive judgments on this when sharing this paper in reading group. I took 1 hour and 40 minutes to introduce this work in details.

Conclusions

Don’t forget to see the conclusions and check whether the contributions are in consistency, and are delivered without a sneaky change in concepts.

Implementations

This is an expansion of the system design. Read this when you have good understanding of the architecture of this paper.

This is pretty much it. When you read a paper, please pay more attention, checking :

  • the consistency of their claimed contribution, be aware of sneaky concept change
  • whether the problem itself is defined according to the system or findings itself. I call it ‘reverse definition’ in that usually you have to make discoveries and have a problem and then come up with a solution, rather than churning out a paper when you have a system.
    • Note that discoveries have to be made before a problem is discovered, and discovery is different from solution. A good paper usually starts from good discovery and then define a novel problem of real impact, and then get a solution. Some papers have a solution first, then tries to modify a problem to fit them to a solutions. If it fits bad, well……
  • whether the paper merely repack some existing techniques and sells the technique just for another problem
  • whether they have really done better than those works criticized by them

and so on.

A good security paper should have good novelty, thoughtful insights, thorough evaluation, appropriate introduction of related work, and real life impacts. A good paper should not only present a good solution, but also insightful introduction for the other researchers. The reviewers in security are indeed picky, but sometimes they may fail to find vulnerabilities in each work. Therefore appearing on a top conference does not guarantee that a paper is truly solid and of high quality. To judge them, only reading technique and experiences of a reader can be beneficial.

Hard as it may be, I still hope that you will enjoy reading paper and doing research–Jesus, compared with the grinding time squeezing out solutions and papers, how relaxing it is to have time to read paper !

And…Nutshell, have fun doing research. :P