Paper Critiques 📝

Back to the Data Engineering Course - 2023 Spring (AI5308/AI4005)

Paper critique is an academic writing that summarizes and gives a critical evaluation of a concept or work. Or, to put it simply, it is no more than a summary and a critical analysis of a specific issue. This type of writing aims to evaluate the impact of the given work or concept in its field.

Three critiques

In each class you’ll read textbook chapters or papers, and sometimes we will ask you to write a short critique for those articles. Critique writing helps you to practice writing. At the same time, it provides opportunities to criticize the (well-written) papers in the field. Of course, no papers are perfect, and there are many rooms for improvement!

Here are the topics for you to write a critique about. Each topic comes with relevant papers.

• Critique 1 (Topic: Using machine learning in a right way)
  • Paper: 150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com
  • Due: Mar 27, 23:59 (KST)
• Critique 2 (Topic: Applying state-of-the-art models in a target domain)
  • Paper: e-CLIP: Large-Scale Vision-Language Representation Learning in E-commerce
  • Due: Apr 24, 23:59 (KST)
• Critique 3 (Topic: Responsible AI - lessons learned on developing DALL-E and GPT-3 at Open AI)
  • Article: DALL·E 2 Preview - Risks and Limitations
  • Article: Lessons learned on language model safety and misuse
  • Due: May 22, 23:59 (KST)

You need to submit the critiques through this form. No late submissions are allowed. After the deadline, you are able to see other people’s critiques and there will be a in-class discussion regarding to your critiques.

Gradings

Critiques take up 15% of your grades. There will be three critiques and each critique will take about 5% of your grade. Critiques will be evaluated by staffs. You will receive one of the following grades:

• Check Plus (5 points) - The critique is very well written. Strength/weakness items are very insightful.
• Check (3 points) - It looks okay. It’s likely that most critiques will belong to this class
• Check Minus (1 points) - The critique is weak. For example, a summary is very much vague. Strength/weakness items were trivials and are not insightful at all. Trivial questions were asked, and discussion was very shallow.
• No submission / late submission (0 points)

Critique Writing Guidelines

Here is a overleaf template to kick-start your writing, copy this project and use for your words.

• Paper Name - write down the name of the paper
• Rating: X out of 5 - rate the current paper (we’ll exclude lowly rated papers in the future)

Provide a summary in your own words (one paragraph)

• Provide a brief description of the results based on your perspectives
• Do not simply repeat the words in the paper, but try to explain things on your words in a less formal way
• Try to summarize the main concept and key contributions
• Add one line comment about whether the contributions are significant and whether the work is effective at delivering the key messages

Strength & Weakness

• You then need to identify several items of strengths and weaknesses. One suggestion is to find three strengths (positive aspects) and one weakness. As a junior researcher, it’s difficult to find weaknesses!
• Strengths could be related to 1) storytelling of problem statements, 2) well summarized related work, 3) great ideas of doing user studies, 4) new methods of data analysis, 5) inspiring discussion, 6) promising research directions
• Weakness could be related to 1) less clear contribution to the community, 2) flaws in study design, 3) lack of generalizability, 4) flaws in evaluation, 5) missing important steps, 6) missing important aspects (in different phases), 7) lack of justifications, 8) weak ecological validity, etc.
• There are many parts of the paper that you can critique. Recall that no paper is perfect; it’s easy to find weaknesses. Likewise, you can easily find strengths in the paper. Several examples include:
  • (strength) “I really like the experiment setup. The authors deployed the system in the wild and recruited around 1000 users! This large scale evaluation clearly shows the ecological validity of the proposed system.”
  • (weakness) “I wish the authors compared the system with other approaches. Currently, the authors showed relative improvements over the naive solutions.”
  • (weakness) “In Section 3, the authors proposed to use the approach X, but that was not well justified. The authors could have used the approach Y instead. I wondered how the results would be if they used the approach Y.”
  • (strength) “The current discussion is very much inspiring. Initially, I thought that the authors talked about a very narrow problem of solving a specific problem. In the discussion, however, the authors explored diverse opportunities about how the current findings can be applied! After reading the discussion, I was convinced that this research area is very important and there should be further studies in this direction.

Questions:

• Parts that you have a hard time understanding
• Something that you want to know more
• Something that is not clear in the paper

Discussion

• After reading the paper, are there any changes in your opinion or perspectives on this topic? If so, state why (previously vs. now)
• Are there any possibilities of using the knowledge learned from this paper in your research project? (For example, “The current evaluation framework is very much related to my project. I’m thinking of adopting the following measures: speed of input and…”)
• Did future research directions or discussion inspire you? If so, state what are they?
• Are there any things that you want to discuss in the class? (For example, “ the authors raised ethical concerns on the approach Y, but I’m not so convinced of that. I would like to hear from others about this aspect.”)

Example Critique:

This is one example borrowed from Prof. Uichin Lee’s CS592 Advanced HCI class at KAIST. It’s a bit longer than he expected, but this example well delivered the key aspects of the critique and it received a “check plus”.

Title: Review spotlight: a user interface for summarizing user-generated reviews using adjective-noun word pairs

Rating: 4/5

Summary: When there are tons of reviews and it’s difficult to quickly extract necessary information from there. The authors came up with an interesting idea of summarizing user-generated reviews based on adjective-noun word pairs (just like “good taste” or “bad Sushi”). Overall, I think that the authors made an interesting contribution to the HCI community in that the authors proposed new ways of quickly navigating a large amount of textual data. This work consists of a formative study followed by system design and evaluation, and system building and evaluation were thoroughly done, well justifying their key ideas.

Strength: (1) I was inspired to know that people consider more when they make a decision among various options, especially during reading user-generated reviews. Someone might focus on the subjective (evaluative) words while others on the objective (descriptive) ones. Through a formative study (with interviews), the authors aimed to summarize user needs related to that.
(2) The authors explained well why they tried to use tag clouds. In related work, they provided limitations of previous studies and related them with where to focus. As a non-expert in this area, it was easy to understand and follow their perspective.

Weakness: One of the main components in the system was using ‘adjective-noun’ word pairs. However, the authors did not explain why they use this pair instead of other combinations. Is it much better or effective than others? How about noun-noun word pairs in each sentence? I wish that the authors provided some discussion about alternative approaches in the discussion section.

Questions: (1) What if people use ‘not’ or ‘no’ in the sentence to make it negative? For example, if the sentence is ‘This place does not provide good food.’ or ‘They used not fresh meat.’ Is it still possible to classify the sentiment well? (2) It is hard to understand the last sentence in the second paragraph of ‘Quantitative Analysis of Review Spotlight Usage.’ Isn’t it too obvious that people explore more in detail since only they can see is the combination of two words? Exploring more in detail could happen in both review pages and Review Spotlight. Was it more effective in letting people explore the detailed information than it was in the review page?

Discussion: (1) Previously I didn’t have any ideas about information overloading. After reading this paper, I realize the importance of data summarization and visualization. When I look at the amazon review, it shows both feature level review ratings as well as what people frequently mentioned. It’s great to see that this kind of data summarization is used in commercial systems. (2) I think that the current system may be vulnerable to gaming behaviors. As people learn how the system works, they might use the same words intentionally to manipulate the reviews.