Uses of video annotation in research

Olivier Aubert - www.olivieraubert.net

AdA project workshop - 17/11/2017

Where do I speak from?

• Research engineer (currently @UnivNantes) / consultant
• Research themes:
  Knowledge engineering - Document engineering - Audiovisual annotation - Hypervideos - Data visualization - Activity traces - User-centered systems
• Lead architect and developer for video-annotation solutions: Advene, COCoNotes, Visu…
• Interdisciplinary projects

Outline

• Some definitions: annotations, active reading…
• Overview of video annotation tools
• Advene focus with illustrating examples
• Conclusion

Annotations in research

Annotation - one of the fundations of scholarship work

Page of the Codex Oxoniensis Clarkianus 39 (Clarke Plato). Dialogue Gorgias. Public Domain

Annotation definition

• annotation - information (free text, constrained vocabulary, structured data, image, graphic…) linked to a document through an anchor
• video annotation - anchor = (spatio)temporal fragment of a video, defined by a begin and end timecode

WebAnnotation definition

Various video annotation communities

Active reading

Advene project

Annotate Digital Video, Exchange on the NEt

to be added to something or become a part of it, though inessential (Webster, 1913)

Create, use, exchange analyses of audiovisual documents as hypervideos

• Started 2002 with Yannick Prié and P.A. Champin
• Free software (GPL) cross-platform desktop app.
• www.advene.org

Anecdote

Mulholland Drive (2001) by David Lynch

• Many interpretations, discussions, clues…
• But…
  • how can we express our analyses and validate them?
  • how can we share them?

No existing software – we had to make one

Scientific motivation

• Knowledge engineering
• Document engineering
• Human-Computer Interfaces
• Datavisualization
• Activity traces

Principle

Interface

Features

• Annotation of any digital video (video file, DVD, stream)
• User-defined structure and visualisations (templates, rules…)
• Preservation of structured annotation information as long as possible.
• Package as bundle (annotations, annotation structure, views, queries) independent from video
• Ability to work offline
• Simple data model (easy data integration)
  • import filters
  • no strict constraints
• Flexible and extensible (through plugins)

Different levels of flexibility

• Accompanying the exploratory analysis
• Variety of annotation creation interfaces
  • Manual annotation with possible assistants
  • Using automatic results as base
• Flexibility in annotation structure
  • Dynamic (re)structuring
  • Allow non-conformant data
• Flexibility in annotation visualization

Existing import filters

Generic import mechanism

• Data: Text, Typed text, lsdvd, Xi, Anvil, ELAN, Subtitle (SRT), PRAAT, CMML, IRI, IRIData, Cinelab, Youtube XML, AnnotationGraph, Final Cut Pro XML, DCP, TurTLe (RDF), MPEG7, Transcriber, TED
• Data processing: NERD (Named Entity Recognition and Disambiguation), Video concept detection
• Movie processing: ShotdetectApp, Sound enveloppe, Audio segmentation, Feature detection (face…), (Speech recognition), Barcode (qr-code) extraction

Output formats

• Basic: through the template language
  MPEG7, RDF, SMIL, SVG, HTML, CSV, json, TimedText, SRT…
• More complex: through plugins (website export)

Application examples

• Conference recording navigation : Tim Berners Lee
• Movie analysis : Nosferatu, Il caso Mattei
• Interview analysis : Museographie
• Lab experiment analysis
• Accessibility for blind users

Example: Nosferatu

Hypervideo publishing

Example: Nosferatu nightmarish scenes

Graphical annotation

Example: Il caso Mattei

Work with Livia Giunti for her PhD about digital tools and time representation in movies

• Importance of digital analysis tools
• Comparison of different tools
• Study of annotation structuring
• Definition of views dedicated to temporality (shot duration bar
• graph, chronologic vs diegetic time)

Example: Il caso Mattei

Dedicated datavisualization

Example: Museographie

Objective: study visitor's perception of museum exhibits through analysis of visitor video interviews

Example: Museographie

Analysis process

• Transcribe interview (note-taking view)
• Identify hexadic signs (from course-of-action methodology)
• Use relations to express courses of experience
• Generate visualizations through templates
• Write analyzes
• Publish on the web

Example: Museographie

Exploratory structuring
Analysis publication as custom hypervideo

Example: Lab experiment analysis

A Case Study of Object and Occlusion Management on the eLabBench, a Mixed Physical/Digital Tabletop (Aurélien Tabard - University Lyon 1/CNRS)

• Graphical annotation of video capture of a laboratory bench
• Study of user behaviour through most used bench areas

Example: Lab experiment analysis

Example: Lab experiment analysis

Example output

Quantitative processing of graphical annotations

Example: accessibility

Dynamic annotation rendering through a Braille table

ADA project developments

• update code to support recent libraries
• improve performance
• interface developments for constrained vocabularies
• improve merging/importing features for collaboration
• bridge for semantic web interoperability (ontology import, rdf export)
• import from HPI concept detection server
• additional visualisations

Summary

• Video annotation is a complex task
  • Genericity vs. simplicity
  • Automation vs. specificity
• Some guiding principles
  • Flexible data structure
  • Various visualizations
  • Allow non-consistent data (but warn about them)
• Annotations are precious
  • need for interoperability in order not to lose data
  • how much can you trust online platforms?

Thank you!

Your questions and remarks are welcome

@Olivier_Aubert
www.olivieraubert.net
www.advene.org

Other contexts

Learning / Pedagogy

COCONotes

Learning / Pedagogy

Visu

Unsorted links

• NBA: http://replay.demo.crankuptheamps.com/ : http://www.crankuptheamps.com/blog/posts/2015/10/01/nba-of-data-science/ (using data from stats.nba.com)
• Visualizations: cinemetrics
• Corpus exploratin: 0xdb, pando.ra
• VATIC: http://carlvondrick.com/vatic/ (online video annotation for using with Mechanical Turk)
• https://help.switch.ch/cast/annotate/ : video annotator integrated into switchcast video management system. Free text or structured annotations.
• http://social.cs.uiuc.edu/projects/vcode.html : (MacOS only) esp. Coder Agreement Support and Coder Conflict Resolution Support
• https://www.theeminentcodfish.com/video-software-review/
• https://github.com/databrary/design/blob/master/references.md#metacollections
• collaboration / web-based (vatic, switchcast)