Hard attention is relatively less explored than soft attention.
This paper showed that hard attention can be competitive and efficient as soft attention by bootstraping hard attention. In constrast to soft attention, hard attention discretely choose the point to attend.
The key idea is to use L2-Norm of the activations of each objects. Vanila Hard Attention Network(HAN) chooses top-k object with the largest L2-Norm. On the other hand, AdaHAN make a threshold to select objects.
Self-attention can be applied to the end of the objects to provide inductive bias for relational reasoning.
HAN achieved competitive result as soft-attention networks.
Subscribe via RSS