COMET: Commonsense Transformers for Automatic Knowledge Graph Construction

Commonsense Knowledge Graph

Knowledge graph is graph representation of knowledge. Entities are represented as nodes and relations between entities are represented as edges. Commonsense knowledge graph stores commonsense knowledge in form of graphs. Two of common dataset for commonsense knowledge graph are ATOMIC and ConceptNet.


image Commonsense knowledge graph is inevitably far from perfect since infinite number of facts can be considered as commonsense. COMET tried to transfer implicit commonsense knowledge from pretrained language model to explicit knowledge graph. More precisely, the model of COMET completes the object entity given subject entity and a relation.


image COMET uses one of the most popular pretrained language model: GPT. Along with BERT, GPT pretrained with huge amount of data is one of the most succuessful natural language backbones based on self-attention mechanism that can be fine-tuned for various other tasks. COMET fine-tunes GPT on entity completion task.

image The input structure changes a little based on dataset, but basically arranged tokens in order of subject entity, relation, and object entity with mask token between them. The goal is to output object entity given subject entity and a relation.

\mathcal{L} = - \sum_{t=|s|+|r|}^{|s|+|r|+|o|} \log P(x_t|x_{<t})


Using pretrained GPT turned out to be effective in relation completion task. The outputs are not only evaluated natural to human, but also are novel. Some of interesting outputs are shown below. image

Bosselut, Antoine, et al. “COMET: Commonsense Transformers for Automatic Knowledge Graph Construction.” arXiv preprint arXiv:1906.05317 (2019).

© 2017. by isme2n

Powered by aiden