Video annotation with Advene

Olivier Aubert - www.olivieraubert.net

@UniBielefeld Digital Academy - 22/09/2022

Where do I speak from?

• Visiting Assistant Professor (Nantes University) / Consultant in Knowledge Engineering
• Research themes:
  Digital Humanities - Knowledge engineering - Document engineering - Audiovisual annotation - Hypervideos - Data visualization - Activity traces - User-centered systems
• Lead architect and developer for video-annotation solutions: Advene, COCoNotes, Visu…

Outline

• Some definitions: annotations, active reading…
• Advene usage
• Illustrating examples
• Conclusion

Annotation in research

Annotation - one of the fundations of scholarship work

Page of the Codex Oxoniensis Clarkianus 39 (Clarke Plato). Dialogue Gorgias. Public Domain

Annotation definition

• annotation - information (free text, constrained vocabulary, structured data, image, graphic…) linked to a document through an anchor
• video annotation - anchor = (spatio)temporal fragment of a video, defined by a begin and end timecode

WebAnnotation definition

Video specificities

• Temporal nature: impossible to get a global/overview of the whole document itself. We have to use representations/surrogates.
• No obvious/trivial segmentation (as word/sentence/paragraph/page in text)
• Cognitive complexity: the time of the media can conflict with the time of the reflection and the time of annotation.

Various video annotation communities

Active reading

Advene project

Annotate Digital Video, Exchange on the NEt

to be added to something or become a part of it, though inessential (Webster, 1913)

Create, use, exchange analyses of audiovisual documents as hypervideos

• Started 2002 with Yannick Prié and P.A. Champin
• Free software (GPL) cross-platform desktop app (Linux/Windows/Mac)
• www.advene.org

Scientific motivation

• Knowledge engineering
• Document engineering
• Human-Computer Interfaces
• Data visualisation
• Activity traces

Principle

Interface

Features

• Ability to work offline
• User-defined structure and visualisations (templates, rules…)
• Package as bundle (annotations, annotation structure, views, queries) independent from video
• Simple data model
  • import filters
  • no strict constraints
• Flexible and extensible (through plugins)

Advene in practice

What will we obtain?

Example package - Source video - template package

Typical process

• Transcribe a video or audio (Note Taking tools, or automatic processing)
• Define analysis categories (in the Timeline)
• Categorise annotations (in the Timeline)
• Navigate/search annotated video
• Generate visualisations

General interface

• Flexible interface
• Different views used according to the task
• 1 default configuration (workspace) saved in the package
• Possibility to save named workspaces (view configurations)

What/how to transcribe?

• Annotating is not only defining content.
• Importance of segmentation of temporal/opaque content
• Difference between time point and interval/annotation
  • Single points (cursory reading, points of interest)
  • Coarse segmentation (chapters, sequences)
  • Precise segmentation in some cases (subtitles/transcription…)

Transcribe a video/audio

Documentation

Basic shortcuts of the Note Taking view:

• Control-space: play/pause
• Control-left: 2 seconds rewind
• Control-Enter: insert a timestamp mark in the transcription

Convert notes to annotations

Notes are just text with timestamp marks in it. Not yet annotations handled by the platform.

Other possibilities

• ActiveBookmark view
  • Insert keyboard shortcut
• Create annotation from the timeline
  • Right-click / Create annotation
  • Control-click in the timeline background
• Toolbar buttons:
  • Create Text annotation
  • Create Graphical annotation

Extract information

Categorise information

Navigate/search

Generate visualisations

Application examples

• Conference recording navigation: Tim Berners Lee
• Movie analysis: Nosferatu, Il caso Mattei
• Interview analysis: Museographie
• Semantic annotation: AdA project
• Lab experiment analysis
• Accessibility for blind users

Example: Nosferatu

Hypervideo publishing

Example: Nosferatu nightmarish scenes

Graphical annotation

Example: Il caso Mattei

Work with Livia Giunti for her PhD about digital tools and time representation in movies

• Importance of digital analysis tools
• Comparison of different tools
• Study of annotation structuring
• Definition of views dedicated to temporality (shot duration bar
• graph, chronologic vs diegetic time)

Example: Il caso Mattei

Dedicated datavisualization

Example: Museographie

Objective: study visitor's perception of museum exhibits through analysis of visitor video interviews

Example: Museographie

Analysis process

• Transcribe interview (note-taking view)
• Identify hexadic signs (from course-of-action methodology)
• Use relations to express courses of experience
• Generate visualizations through templates
• Write analyzes
• Publish on the web

Example: Museographie

Exploratory structuring
Analysis publication as custom hypervideo

Exemple: AdA project

https://projectada.github.io/

• Affektrhetoriken des Audiovisuellen - Audio-visual rhetorics of affect
• Systematic-empirical study of staging patterns in audiovisual representations of the 2007 financial crisis
• Interdisciplinary collaboration: Cinepoietics (Media Studies, Freie Universität Berlin), HPI Potsdam (semantic web, video analysis)

AdA project - Challenges and needs

Challenges

• Developing a methodological framework for movie analysis, with quantitative support
• Semantic video-annotations
• Exploration of automation possibilities (feature extraction)

Needs

• Provide a 2-way bridge between users and semantic data
• Optimize the manual annotation process
• Build a ground truth for future automation

AdA project - Advene screenshot

Principle

Semantic web extensions

• WebAnnotation/JSON-LD import and export
• OWL importer
  • Convert OWL schema into Advene structures
• OWL-derived metadata preserved
  • used in Checker framework
  • used for GUI shortcuts
  • used in WebAnnotation export

Produced ontology and data

https://projectada.github.io/

Feature-extraction plugins

• Using Advene generic feature-extraction framework or API import filters
• Principle
  • define a gstreamer audio/video decoding pipeline, such as videoconvert ! videoscale ! video/x-raw,width=160,pixel-aspect-ratio=(fraction)1/1,format=GRAY8
  • define a process_frame method, called for each frame
  • produce annotations as python dicts

Example plugin outputs

Visualisation

• Multiple visualisations are already present in Advene: timeline, table, transcript…
• Advene embeds a webserver and a template language interpreter, able to generate and serve X/HTML documents
  • Agile testbed for experiments

Web-based timeline

visualization

One possible visualisation, overview of annotations for one single movie

Example: Lab experiment analysis

A Case Study of Object and Occlusion Management on the eLabBench, a Mixed Physical/Digital Tabletop (Aurélien Tabard - University Lyon 1/CNRS)

• Graphical annotation of video capture of a laboratory bench
• Study of user behaviour through most used bench areas

Example: Lab experiment analysis

Example: Lab experiment analysis

Example output

Quantitative processing of graphical annotations

Example: accessibility

Dynamic annotation rendering through a Braille table

Summary

• Video annotation is a complex task
  • Genericity vs. simplicity
  • Automation vs. specificity
• Some guiding principles
  • Flexible data structure
  • Time segmentation/navigation
  • Various visualizations
  • Allow non-consistent data (but warn about them)
• Annotations are precious
  • need for interoperability in order not to lose data
  • how much can you trust online platforms?

Thank you!

Your questions and remarks are welcome

@Olivier_Aubert
www.olivieraubert.net
www.advene.org