Describing Common Human Visual Actions in Images BMVC'15

You may search for interactions by specifying a query composed of:

• Visual Action: any of the Visual VerbNet actions (having unique and unambiguous visual connotation), acted by a subject upon a certain object, as explained here.
• Visual Adverb: a property associated to a Visual Action, such as the relative location or distance between subject and object, or to the subject performing the action, such as its emotional state.
• Object Category: any of the 81 object categories annnotated in the MS COCO dataset, on which a certain Visual Action is being performed.

You may also submit a query made of a combination of the three above. This will retrieve all the annotations (if present) containing a certain action, with a certain property, acted upon a specific object.

There are two options for composing the desired query:

1) Manual Selection

• Click on a Button depending on the query element you would like to select.
• Search for the element (or elements) within the specified categories and manually select it.
• When clicked upon the element will become highlighted for confirmation and automatically appear in the search bar.

2) Search Bar Autocomplete

• Start typing in the search bar for a Visual Action, Adverb or Object Category.
• All the entries in Visual VerbNet and Object entries having a common root with the typed word will appear and become selectable.
• Once a valid word is entered it will be added to the search bar and a new search can be performed.

Once you composed your query and pressed the Search button you will be presented with results retrieved from the coco-a training set, if present. The results are organized in the following elements:

1) Number of retrieved annotations

The current number of annotations in the coco-a training set matching the selected query.

2) Image Id Button

Hover the mouse on top of this button to view the original image from the MS COCO dataset without the subject/object interaction annotations.

3) Give Feedback Button

Click on this button to give feedback about a specific annotation. Please use it to report errors, inappropriate material or simply doubts and suggestions regarding an interaction annotation.

4) Annotation Results

Each image represents a single interaction or solo action annotation, and can contain either a subject (highlighted in blue) and an object (highlighted in green), or simply a subject if the annotation represents a solo action.

Each interaction can thus be composed of two parts:

• Subject properties: describe the subject's state and solo actions, independently from the interaction carried out with the object (if present). Hover the mouse inside the subject's highlight in the image to view these annotations.
• Interaction properties: describe the subject's interaction specifically with the highlighted object and properties of this interaction, such as the relative location and distance. Hover the mouse inside the object's highlight in the image to view these annotations.

We selected three interesting sample queries to show all the functionalities explained in the previous sections.