MPI-SP Seminar - Sundong Kim (GIST)

Date: Tuesday, 25 August 2026 · MPI-SP, Data Science Group (Host: Meeyoung Cha)

Title

How My Problems Kept Getting Redefined: From Customs Fraud to ARC-AGI-3

Speaker

Sundong Kim (Assistant Professor, GIST)

Abstract

Look closely at a problem many people are working on, and something important has usually been left out of how it was posed — and that excluded axis is often where the interesting work turns out to be. This talk follows how the problems I work on kept getting redefined: a PhD problem posed as classification when it was really survival regression; a customs fraud detection system that reality forced to grow five new axes; and ARC-AGI, whose sealed test set structurally excluded the memorization-based networks that were winning everywhere else, restoring the axis of time spent thinking once a problem arrives. ARC-AGI-2 then put the price of that thinking into the score, and ARC-AGI-3 moved the target again, rewarding systems that adapt to something unfamiliar with as little interaction with the environment as possible. Each restoration did more than change a number; each ruled out a class of methods that had been working perfectly well until then. I would like to close my talk with the final question: now that an agentic framework solves most of what we put in front of it, which problem setup is still worth posing?

Bio

Sundong Kim is an Assistant Professor at GIST AI. His research centers on abstraction and reasoning: how to evaluate it, and how to build agents that acquire it, with a long-standing focus on the ARC-AGI benchmark as a testbed for human-like abstract reasoning. His group has built a series of ARC-related environments and datasets, including O2ARC, ARCLE, ARCTraj, and GIFARC, and now works on ARC-AGI-3, developing self-evolving agents with world-model-based scaffolds. Before joining GIST in 2022, he was a Young Scientist Fellow at the Institute for Basic Science, hosted by Mia Cha, where he led a three-year collaboration with the World Customs Organization on customs fraud detection and AI for social good. He received his PhD from KAIST in 2019.

Web: https://sundong.kim

This is the long version of a talk I gave at the Max Planck Institute for Security and Privacy on 25 August 2026.

Two things I have come to believe

Look closely at a problem many people are working on, and something important has usually been left out of how it was posed. Not hidden — simply absent, because the setup everyone inherited had no slot for it. That excluded axis is, in my experience, where the interesting work turns out to be.

The second thing took me longer to notice. Putting the axis back does not merely make the numbers more honest. It changes which methods are still admissible. Approaches that had been working perfectly well stop being candidates, because they were only ever viable in the version of the problem that left the axis out.

Almost everything I have worked on for the past ten years is an instance of those two sentences. What follows is the sequence, in order.

1. The axis I did not know was missing

My PhD was on customer revisit prediction. Retail analytics companies had begun deploying Wi-Fi sensors inside stores, and we had indoor mobility data from seven flagship stores in Seoul — between 220 and 990 days of collection each, and anywhere from 110 thousand to two million visits per store. The question was whether a customer’s movement inside the store today tells you whether they will come back.

I posed it as binary classification. I built ten groups of handcrafted features — dwell time, travel distance and speed, area preference, entrance and exit patterns, store accessibility, group movement, time of visit, upcoming events — and trained gradient boosting on top of them. That work became Utilizing In-Store Sensors for Revisit Prediction (ICDM 2018), later extended into a journal version in Knowledge and Information Systems.

The setup had a hole in it, and the hole was time.

Revisit is not a property of a visit. It is an event with a date attached. When the observation window closed, a customer who had not yet come back was not a negative example — they were a partial observation. In our data, more than seventy percent of visits came from first-time visitors, and their revisit rate was around fifteen percent, which means the majority of the dataset sat on the axis we had deleted, and sat there mislabelled. In the classification framing those visits were either dropped as missing data or silently counted as “will not return.”

I want to be careful about why this happened, because the honest answer is not that anyone was careless. Every request that shaped the work was a reasonable one:

  • Reduce it to a binary problem and get F1 above eighty.
  • Split seventy–thirty and compare against the baselines.
  • Reviewers say the model is dated; put a neural network on it.

The first invites class imbalance and deletes the event time. The second introduces data leakage in a temporal setting, because a random split lets the future inform the past when the whole point is that training precedes testing. The third does not touch the problem definition at all — it is a request about method. Three reasonable asks, and not one of them was about the shape of the question. Goodhart’s law had arrived early: the proxy was F1 on a downsampled binary task, and once the proxy becomes the target, whatever the proxy does not contain stops existing.

After my proposal I went back and rebuilt it. SurvRev, published as Revisit Prediction by Deep Survival Analysis (PAKDD 2020), encodes each visit with a low-level module over the indoor trajectory, aggregates a customer’s visit history, and emits a predicted event rate over the next k days rather than a single interval. The three things it was designed to survive are exactly the three the earlier framing had ignored: class imbalance, data leakage in temporal settings, and censored data.

And here is the part that mattered more to me than the model.

Restoring the axis did not shuffle the ranking. It inverted it. Evaluated on censored customers with the concordance index, gradient boosting came out at 0.420 on one store — below chance. Evaluated on new customers by F-score, the majority baseline scored exactly 0.000, because under the old framing “predict that nobody returns” was a serviceable strategy and under the new one it is not a strategy at all. Even the metric had to change: accuracy was no longer a coherent quantity, so we moved to C-index and to F-score restricted to the population the old setup had discarded.

That is the second law in its first appearance. The method that had been winning was not somewhat worse under the corrected definition. It was disqualified.

A footnote that keeps me humble

Around that time I spoke with people doing related work in industry. They were polite, but it was clear they found the setup a toy. They were not predicting a binary; they had click-through, purchase, price, inventory, and a deployment loop feeding back into the data. I had restored one axis. They had restored several, because they owned the pipeline that produced the data and I did not — our sensor logs simply contained no purchase information, so there was nowhere further for me to climb.

I think about this more often than about the award. Restoring an excluded axis is a ladder, not a switch. There is no correct problem definition waiting at the top. There is only a definition that excludes less than the previous one, and how far you get is bounded by what you have access to.

2. Five rounds at the border

Customs came next, and the reason was not noble. Customs fraud detection is anomaly detection over a temporal stream of declarations, and I had just spent two years learning that time is the axis people delete. Once you have restored an axis, it starts shaping which problem you pick next.

What happened over the following years was different from the PhD, though. There, I found the missing axis myself, late. In the collaboration with the World Customs Organization, reality handed me one round after round, and each round had the same shape.

Round 1 — revenue. Intentional manipulation of invoices, leading to undervaluation of traded goods, is the most common way to avoid ad valorem duties. We built DATE, a tree-aware dual attentive model that scores import declarations, and reported precision of 92.7% on illegal cases together with recall of 49.3% on revenue while inspecting only one percent of all trade flows (DATE: Dual Attentive Tree-aware Embedding for Customs Fraud Detection, KDD 2020). That second number is the point. An administration does not only want to catch fraud; it wants to recover duty, and precision and recall on illicit cases cannot express that. Once revenue entered the objective, a single risk ranking was no longer sufficient and the model became dual-task.

Round 2 — what we would fail to learn. The system worked, and then it degraded, for a structural reason: labels exist only for the declarations you inspect. A selection system that inspects only what it is already confident about progressively blinds itself, and the drop shows up a year later, in someone else’s evaluation window. The excluded axis was everything outside the current model’s field of view. Purely exploitative selection was disqualified; we moved to a hybrid strategy that spends part of the inspection budget on exploration (Active Learning for Human-in-the-Loop Customs Inspection, IEEE TKDE).

Round 3 — drift, and that it differs by administration. We then had to decide how much to explore, and the answer was not a constant. Declaration patterns drift, and they drift at different rates in different countries, so the right exploration ratio in one administration is wrong in another. A fixed rate was disqualified; the ratio had to be set adaptively (Customs Fraud Detection in the Presence of Concept Drift, ICDMW 2021).

Round 4 — the law. The administrations that most needed a strong model had the least data, and the obvious fix — pool it — was unavailable, because cross-border data sharing is legally constrained. This is an axis no benchmark would ever contain and it is completely binding in practice. Pooling was disqualified, so we shared by-products of the models instead, through domain adaptation (Knowledge Sharing via Domain Adaptation in Customs Fraud Detection, AAAI 2022).

Round 5 — the officer. For product classification, predicting the right HS code turned out to be the easy half. An officer who must justify a reclassification to a trader needs the supporting passage from the explanatory notes, not a confident number. A black-box classifier was disqualified (Explainable Product Classification for Customs, ACM TIST 2024). In beta testing with the Korea Customs Service, the comment that stayed with me was that the tool was useful for training new officers — which is only true of a system that shows its reasons.

Five rounds, five axes, five classes of method ruled out. What those years left me was not five papers. It was the reflex.

3. What to measure

I work on abstraction and reasoning now, and in this field the same move has a name: evaluation design.

The clearest case I know is ARC-AGI. It is easy to describe it as a set of coloured grid puzzles and miss what made the first version consequential, which was a policy rather than a task: the test set stayed sealed.

Consider what the field was optimizing in 2019. Segmentation, classification, detection — benchmarks on which a network that has absorbed enough of the distribution in advance can answer in a single forward pass, and where absorbing the distribution is not a shortcut, it is the method. Against a sealed set of genuinely novel tasks, that strategy has nowhere to go. Memorize-then-infer was not merely weaker on ARC. It was structurally excluded.

What the benchmark had quietly restored is the axis I care about most now: time spent thinking once the problem arrives. I would argue that a good part of what followed — the turn toward systems that deliberate before answering — came from a generation of models having to earn their answers inside that window.

That definition still left things out, and my lab has spent five years on what it left out.

How a solution is reached, not only what it is. At the time, nearly everyone treated ARC as generation: map the input grid to the output grid, end to end. We treated it as something solved step by step, closer to program synthesis, which meant we needed data about steps rather than answers. So we built a web platform where people solve and create ARC tasks through discrete operations, with the whole trajectory recorded (O2ARC 3.0, IJCAI 2024 demo), released the resulting human reasoning trajectories as a dataset and benchmark (ARCTraj, KDD 2026), and wrapped the environment for reinforcement learning (ARCLE, CoLLAs 2024). The excluded axis was the human solution process. You cannot study it with a dataset that stores only final grids.

Whether the evaluation itself is contaminated. When large language models began posting strong reasoning numbers, we asked what those numbers were made of. Borrowing the language of thought hypothesis, we decomposed reasoning into logical coherence, compositionality and productivity and measured each separately (Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus, ACM TIST 2025). The picture was much less flattering than the headline accuracy. The sharpest result was on consistency: take the tasks a model solved, regenerate fresh input–output pairs from the underlying program, and ask again. More than half the tasks fell below ten percent accuracy. Because ARC grids are discrete and the ground-truth programs are executable, the regenerated pairs are provably not the memorized ones, so pattern matching cannot survive the transfer while program induction can. A benchmark you can find on the internet cannot separate reasoning from recall. Until you restore that axis, you do not know what your score means.

Where the compute is spent. Through 2024 the dominant strategy in the ARC community was augmentation at pretraining time. We bet on the other end — extrapolation at test time — and worked on treating solving as sequential decision-making in a learned latent space, using diffusion-based offline RL over ARCLE (Diffusion-Based Offline RL for Improved Decision-Making in Augmented ARC Task, and the follow-up On the Role of Proposal Support in Diffusion-Based Offline RL for Sequential Decision-Making). In parallel we explored the same problem as abduction: induce candidate programs from each input–output pair, then generalize across them (Abductive Symbolic Solver on Abstraction and Reasoning Corpus). At the time this looked like a side bet against the prevailing view that a large enough pretrained model would simply absorb the task. It has aged reasonably well.

Where the priors come from. ARC states its priors — objectness, counting, elementary geometry — but its tasks are still authored by hand, which means the analogies a solver must find are the analogies one designer happened to imagine. So we built GIFARC, which derives tasks from animated GIFs so that the underlying analogies are ones people already recognize from the physical world (GIFARC: Synthetic Dataset for Leveraging Human-Intuitive Analogies to Elevate AI Reasoning). The excluded axis here is where a prior comes from: a benchmark can declare its priors and still leave open whether they are the ones humans actually reason with.

Meanwhile the benchmark itself kept restoring axes. ARC-AGI-2 put the price of an answer into the score, which is not a footnote: once cost is part of the definition, an unbounded ensemble that brute-forces a solution is no longer a valid answer, however accurate it is. ARC-AGI-3 went further and began scoring how much interaction with the environment a system needs — wasted actions relative to a human.

I want to record one thing about that last step, because it has been the most encouraging experience I have had as a PI. In the spring of 2025 I set a homework in my AGI course that asked students to build an Othello agent for a board they had never seen, with sixty seconds to adapt, and any approach allowed — meta-RL, MCTS, evolutionary search. The point was to strip away everything that could be prepared in advance. That homework became The Expanded Othello AI Arena: Evaluating Intelligent Systems Through Constrained Adaptation to Unseen Conditions (TMLR 2026). A few months later the ARC-AGI-3 preview arrived with a framing not far from what we had converged on.

That is not a claim of priority. It is evidence for something better. Problem definitions are not a matter of taste; they can be discovered. Two groups working independently arrived at the same missing axis, which is reason to believe the axis was really there.

And then the second law fired again, right on cue. Once action efficiency is part of the definition, online reinforcement learning — which pays for competence in episodes — stops being admissible. You need something that can rehearse instead of spend. Which is why our current work builds an executable world model and does its exploring inside it.

4. What we still cannot measure

Which brings me to the part where I do not have answers.

Evaluations that evolve. A great deal of expert trajectory data is being collected right now: the minimal solutions from code golf competitions, and, at a very different scale, the work logs large companies are recording from their own employees. All of it is gathered to grow better solvers. But look at what such a corpus actually is — a growing set of tasks, each paired with the best solution anyone currently has for it. That is also the raw material for a better answer key. The evolution of an evaluation is the evolution of its tasks plus the evolution of the solutions it measures against. Can we build an evaluation that keeps moving, instead of freezing on the day we ship it?

Whether an agent holds to its own principles. We measure whether an agent succeeds. We do not really measure whether it remains the same agent while succeeding — whether the commitments it stated at step one still hold at step fifty, when they have become inconvenient.

Whether it has anything like a drive to persist. One of my undergraduates built a factorial benchmark probing functional self-preservation drive in language models. I am making no claim about inner life. I am saying that “does this system act so as to remain in play” is a measurable property that standard benchmarks do not contain, and that as agents are given longer horizons it will stop being an exotic question.

Whether the environment can move too. Every benchmark described above holds the environment fixed and lets the learner improve. Unsupervised environment design inverts that, generating the environments that are most useful to learn from next, so that the curriculum co-evolves with the student (TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design, ICLR 2026).

The question I actually want to discuss

Here is where I get stuck, and it is why I would rather end with a question than a summary.

An agentic framework now solves a large fraction of what we put in front of it. Not perfectly, but well enough that “can a system do X” is increasingly a question about scaffolding rather than about capability. If that is right, then the scarce resource is no longer the method. It is the problem setup.

So: which problem setups are still worth posing?

I do not think the answer is “harder ones.” Difficulty is easy to manufacture and tells you little. I suspect the answer is setups that exclude less — that force something into the definition which current systems can route around today because nobody is looking. Cost was one such axis, and it changed the field. Interaction budget was another.

I do not know what the next one is. If you have a candidate, I would like to hear it.

Papers referred to above

  • Utilizing In-Store Sensors for Revisit Prediction — IEEE ICDM 2018
  • A Systematic Framework of Predicting Customer Revisit with In-Store Sensors — Knowledge and Information Systems 2020
  • Revisit Prediction by Deep Survival Analysis — PAKDD 2020
  • DATE: Dual Attentive Tree-aware Embedding for Customs Fraud Detection — KDD 2020
  • Active Learning for Human-in-the-Loop Customs Inspection — IEEE TKDE
  • Customs Fraud Detection in the Presence of Concept Drift — ICDMW 2021
  • Knowledge Sharing via Domain Adaptation in Customs Fraud Detection — AAAI 2022
  • Explainable Product Classification for Customs — ACM TIST 2024
  • O2ARC 3.0: A Platform for Solving and Creating ARC Tasks — IJCAI 2024
  • ARCLE: The Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learning — CoLLAs 2024
  • ARCTraj: A Dataset and Benchmark of Human Reasoning Trajectories for Abstract Problem Solving — KDD 2026
  • Abductive Symbolic Solver on Abstraction and Reasoning Corpus — IJCAI Workshop 2024
  • Diffusion-Based Offline RL for Improved Decision-Making in Augmented ARC Task
  • On the Role of Proposal Support in Diffusion-Based Offline RL for Sequential Decision-Making
  • GIFARC: Synthetic Dataset for Leveraging Human-Intuitive Analogies to Elevate AI Reasoning
  • Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus — ACM TIST 2025
  • The Expanded Othello AI Arena: Evaluating Intelligent Systems Through Constrained Adaptation to Unseen Conditions — TMLR 2026
  • TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design — ICLR 2026

The ARC line is the work of my students at the GIST Data Science Lab; the customs line was a collaboration with the World Customs Organization, the Nigeria Customs Service, and colleagues at IBS.