Uses of video annotation in research

Sorry, your browser does not support SVG.

Olivier Aubert -
AdA project workshop - 17/11/2017

Where do I speak from?

  • Research engineer (currently @UnivNantes) / consultant
  • Research themes:
    Knowledge engineering - Document engineering - Audiovisual annotation - Hypervideos - Data visualization - Activity traces - User-centered systems
  • Lead architect and developer for video-annotation solutions: Advene, COCoNotes, Visu…
  • Interdisciplinary projects


  • Some definitions: annotations, active reading…
  • Overview of video annotation tools
  • Advene focus with illustrating examples
  • Conclusion

Annotations in research

Annotation - one of the fundations of scholarship work


Page of the Codex Oxoniensis Clarkianus 39 (Clarke Plato). Dialogue Gorgias. Public Domain

Annotation definition

  • annotation - information (free text, constrained vocabulary, structured data, image, graphic…) linked to a document through an anchor
  • video annotation - anchor = (spatio)temporal fragment of a video, defined by a begin and end timecode

WebAnnotation definition

annotation - information body (free text, constrained vocabulary, structured data, image, graphic…) linked to a target document through a selector

Sorry, your browser does not support SVG.

video annotation - selector = (spatio)temporal fragment of a video, defined by a MediaFragment URI

Various video annotation communities

media studies
Advene - Mediascope - Lignes de Temps

music studies
Travis, MusiqueLab

Anvil - ELAN - Exmeralda - Transcriber

ethology/behavioral studies
BORIS - VCode - Noldus/Observer - NVivo

Amara - AegisSub

COCoNotes - Travis-Go - Lacuna stories - Mediathread

social annotations
Vialogues - TurboNote - Reclipped

computer science
vatic - LabelMe

LongoMatch - Powerchalk - CoachLogic - SportsCode

Active reading

Sorry, your browser does not support SVG.

Advene project

Annotate Digital Video, Exchange on the NEt

to be added to something or become a part of it, though inessential (Webster, 1913)

Create, use, exchange analyses of audiovisual documents as hypervideos

  • Started 2002 with Yannick Prié and P.A. Champin
  • Free software (GPL) cross-platform desktop app.


Mulholland Drive (2001) by David Lynch

  • Many interpretations, discussions, clues…
  • But…
    • how can we express our analyses and validate them?
    • how can we share them?

No existing software – we had to make one

Scientific motivation

  • Knowledge engineering
  • Document engineering
  • Human-Computer Interfaces
  • Datavisualization
  • Activity traces


Sorry, your browser does not support SVG.




  • Annotation of any digital video (video file, DVD, stream)
  • User-defined structure and visualisations (templates, rules…)
  • Preservation of structured annotation information as long as possible.
  • Package as bundle (annotations, annotation structure, views, queries) independent from video
  • Ability to work offline
  • Simple data model (easy data integration)
    • import filters
    • no strict constraints
  • Flexible and extensible (through plugins)

Different levels of flexibility

  • Accompanying the exploratory analysis
  • Variety of annotation creation interfaces
    • Manual annotation with possible assistants
    • Using automatic results as base
  • Flexibility in annotation structure
    • Dynamic (re)structuring
    • Allow non-conformant data
  • Flexibility in annotation visualization

Existing import filters

Generic import mechanism

  • Data: Text, Typed text, lsdvd, Xi, Anvil, ELAN, Subtitle (SRT), PRAAT, CMML, IRI, IRIData, Cinelab, Youtube XML, AnnotationGraph, Final Cut Pro XML, DCP, TurTLe (RDF), MPEG7, Transcriber, TED
  • Data processing: NERD (Named Entity Recognition and Disambiguation), Video concept detection
  • Movie processing: ShotdetectApp, Sound enveloppe, Audio segmentation, Feature detection (face…), (Speech recognition), Barcode (qr-code) extraction

Output formats

  • Basic: through the template language
    MPEG7, RDF, SMIL, SVG, HTML, CSV, json, TimedText, SRT…
  • More complex: through plugins (website export)

Application examples

  • Conference recording navigation : Tim Berners Lee
  • Movie analysis : Nosferatu, Il caso Mattei
  • Interview analysis : Museographie
  • Lab experiment analysis
  • Accessibility for blind users

Example: Nosferatu

Hypervideo publishing


Example: Nosferatu nightmarish scenes

Graphical annotation


Example: Il caso Mattei

Work with Livia Giunti for her PhD about digital tools and time representation in movies

  • Importance of digital analysis tools
  • Comparison of different tools
  • Study of annotation structuring
  • Definition of views dedicated to temporality (shot duration bar
  • graph, chronologic vs diegetic time)

Example: Il caso Mattei

Dedicated datavisualization


Example: Museographie

Objective: study visitor's perception of museum exhibits through analysis of visitor video interviews


  • visitors interviewed while replaying a recording of their visit with subjective camera
  • 40+ annotated videos
  • online publication of views and raw data
  • dedicated visualization
  • automated tasks (via plugins)

Example: Museographie

Analysis process

  • Transcribe interview (note-taking view)
  • Identify hexadic signs (from course-of-action methodology)
  • Use relations to express courses of experience
  • Generate visualizations through templates
  • Write analyzes
  • Publish on the web

Example: Museographie

Exploratory structuring
Analysis publication as custom hypervideo


Example: Lab experiment analysis

A Case Study of Object and Occlusion Management on the eLabBench, a Mixed Physical/Digital Tabletop (Aurélien Tabard - University Lyon 1/CNRS)

  • Graphical annotation of video capture of a laboratory bench
  • Study of user behaviour through most used bench areas

Example: Lab experiment analysis


Example: Lab experiment analysis

Sorry, your browser does not support SVG.

Example output

Quantitative processing of graphical annotations


Example: accessibility

Dynamic annotation rendering through a Braille table


ADA project developments

  • update code to support recent libraries
  • improve performance
  • interface developments for constrained vocabularies
  • improve merging/importing features for collaboration
  • bridge for semantic web interoperability (ontology import, rdf export)
  • import from HPI concept detection server
  • additional visualisations


  • Video annotation is a complex task
    • Genericity vs. simplicity
    • Automation vs. specificity
  • Some guiding principles
    • Flexible data structure
    • Various visualizations
    • Allow non-consistent data (but warn about them)
  • Annotations are precious
    • need for interoperability in order not to lose data
    • how much can you trust online platforms?

Thank you!

Your questions and remarks are welcome


Other contexts

Learning / Pedagogy



Learning / Pedagogy



Unsorted links