about
Treating the Intent Detection Problem as Dynamics in a Low-Dimensional Space (arxiv.org)
1 point by PaulHoule on Aug 19, 2024 | hide | past | pdf | discuss on HN

In plain words: Recurrent networks that label query intent can be read as pushing sentences along a path in a small hidden space toward regions aligned with output labels. Instead of a black box, the paths settle into a few attractors that pull nearby paths in.

Abstract · Interpretation of the Intent Detection Problem as Dynamics in a Low-dimensional Space

Intent detection is a text classification task whose aim is to recognize and label the semantics behind a users query. It plays a critical role in various business applications. The output of the intent detection module strongly conditions the behavior of the whole system. This sequence analysis task is mainly tackled using deep learning techniques. Despite the widespread use of these techniques, the internal mechanisms used by networks to solve the problem are poorly understood. Recent lines of work have analyzed the computational mechanisms learned by RNNs from a dynamical systems perspective. In this work, we investigate how different RNN architectures solve the SNIPS intent detection problem. Sentences injected into trained networks can be interpreted as trajectories traversing a hidden state space. This space is constrained to a low-dimensional manifold whose dimensionality is related to the embedding and hidden layer sizes. To generate predictions, RNN steers the trajectories towards concrete regions, spatially aligned with the output layer matrix rows directions. Underlying the system dynamics, an unexpected fixed point topology has been identified with a limited number of attractors. Our results provide new insights into the inner workings of networks that solve the intent detection task.

Eduardo Sanchez-Karhunen, Jose F. Quesada-Moreno, Miguel A. Gutiérrez-Naranjo
arXiv:2408.02838 · cs.LG, cs.CL, stat.ML · submitted Aug 5, 2024
abstract · pdf · html · Camera-Ready version. Accepted paper at 27th European Conference on Artificial Intelligence (ECAI-2024)

add comment on HN