Advene, 20 years of video annotation

Sorry, your browser does not support SVG.

Olivier Aubert -
LS2N - Nantes Université
Reimagining Annotation for Multimodal Cultural Heritage - 08/02/2024


  • Context: annotations
  • Advene - principles and experiments
  • Lessons learned

Annotations in research

Annotation - one of the fundations of scholarship work


Page of the Codex Oxoniensis Clarkianus 39 (Clarke Plato). Dialogue Gorgias. Public Domain

More recent annotations


CC Public Domain

Now digital…


Various video annotation communities

media studies
Advene - VIAN - Mediascope - (Lignes de Temps) -

COCoNotes - Travis-Go - Mediathread - Mediate - Celluloid

music studies
Travis - (MusiqueLab) - Transcribe

performing arts
MotionBank - MemoRekall

Anvil - ELAN - Exmeralda - Transcriber

behavioral research
BORIS - Transana - VCode - Noldus/Observer - NVivo

Amara - AegisSub

social annotations
(Vialogues) - YiNote - Reclipped

vatic - LabelMe

LongoMatch - CoachLogic - SportsCode

Ecology of hypertext annotation


Advene project

Annotate Digital Video, Exchange on the NEt

to be added to something or become a part of it, though inessential (Webster, 1913)

Create, use, exchange analyses of audiovisual documents as hypervideos

  • Started 2001 with Yannick Prié and P.A. Champin
  • Free software (GPL) cross-platform desktop app.



Scientific motivation

  • Knowledge engineering
  • Document engineering
  • Human-Computer Interfaces
  • Datavisualization
  • Activity traces

Scratch an itch…

Mulholland Drive (2001) by David Lynch


Advene focus

  • Active reading / close reading
  • Single video
  • Companion to exploratory research
  • Flexibility (structures, contents, outputs…)
  • Offline use


Sorry, your browser does not support SVG.

Different levels of flexibility

to accompany the exploratory analysis

  • Variety of supported video formats (video file, DVD, stream)
  • Flexibility in annotation structure
    • Dynamic (re)structuring
    • Allow non-conform data (but warn)
  • Variety of annotation creation interfaces
    • Manual annotation with possible assistants
    • Using automatic results as base
  • Flexibility in annotation visualization
  • Extensibility through plugins

Projects timeline

  • Incorporactions (2004-2007)
  • Cinelab (2007-2008)
  • CineCast (2009-2011)
  • datadata (2010-2011)
  • ACAV (2009-2012)
  • Il Caso Mattei (2010-2011)
  • Museographie (2011-2013)
  • SpectacleEnLignes (2011-2015)
  • (CocoNotes) (2013-2016)
  • AdA project (2017-2021)


  • Lab experiment analysis (SVG annotations)
  • Accessibility for blind users (Braille display)
  • Movie analysis : Il caso Mattei (graphs), AdA (input/ontologies)
  • Interview analysis : Museographie (structure/hypervideo)

Example: Lab experiment analysis

A Case Study of Object and Occlusion Management on the eLabBench, a Mixed Physical/Digital Tabletop (Aurélien Tabard - University Lyon 1/CNRS)

  • Graphical annotation of video capture of a laboratory bench
  • Study of user behaviour through most used bench areas

Example: Lab experiment analysis

Sorry, your browser does not support SVG.

Lab experiment - Example output

Quantitative processing of graphical annotations


Example: accessibility

Dynamic annotation rendering through a Braille display


Example: Il caso Mattei

Work with Livia Giunti for her PhD about digital tools and time representation in movies

  • Importance of digital analysis tools
  • Comparison of different tools
  • Study of annotation structuring
  • Definition of visualisations dedicated to temporality (shot duration bargraph, chronologic vs diegetic time)

Example: Il caso Mattei

Dedicated datavisualization


Example: Museographie

Objective: study visitor's perception of museum exhibits through analysis of visitor video interviews


  • visitors interviewed while replaying a recording of their visit with subjective camera
  • 40+ annotated videos
  • online publication of views and raw data
  • dedicated visualization
  • automated tasks (via plugins)

Example: Museographie

Analysis process

  • Transcribe interview (note-taking view)
  • Identify hexadic signs (from course-of-action methodology)
  • Use relations to express courses of experience
  • Generate visualizations through templates
  • Write analyzes
  • Publish on the web

Example: Museographie

Exploratory structuring
Analysis publication as custom hypervideo


Example: AdA project

  • Affektrhetoriken des Audiovisuellen - Audio-visual rhetorics of affect
  • Systematic-empirical study of staging patterns in audiovisual representations of the 2007 financial crisis
  • Interdisciplinary collaboration with Cinepoietics (Media Studies, Freie Universität Berlin) and HPI Potsdam (semantic web, video analysis)

AdA project - Challenges and needs


  • Developing a methodological framework for movie analysis, with quantitative support
  • Semantic video-annotations
  • Exploration of automation possibilities (feature extraction)


  • Provide a 2-way bridge between users and semantic data
  • Optimize the manual annotation process
  • Build a ground truth for future automation

AdA project - Advene screenshot


AdA project - Principle

Sorry, your browser does not support SVG.

ADA project - Developments

  • maintain code to support recent libraries (Python3 / Gtk3)
  • improve performance
  • interface developments for constrained vocabularies
  • improve merging/importing features for collaboration
  • bridge for semantic web interoperability (ontology import, rdf export)
  • import from HPI concept detection server
  • additional visualisations (web timeline)


  • The technology pharmakon
  • The maintenance burden
  • Usability matters

Technology pharmakon

  • Technology is a requirement in Digital Humanities - epistemic change
  • Technology is moving and fragile
  • Technical debt can accumulate

Video and technology

  • Video material does not exist without technology
  • Initial design decision to use DVD as main source of shared, legit video
    • Simple player abstraction, handling mainly time dimension
  • In 2001: video not first class citizen on the web
    • <video> tag - initial draft in 2007, official W3C recommandation in 2014
    • in web context, use of other tools (flash - deprecated in 2017)
  • What about 360 video? VR experiences?

Annotation standards

  • Many initiatives in 2001 (Mosaic, Annotea…)
  • WebAnnotation W3C working group started in 2014 (after OpenAnnotation)
  • WebAnnotation standardized in 2017
  • For preceding models, model adaptation may be needed

The web browser as universal platform?

  • The browser has become the de-facto universal runtime…
  • …but beware of small discrepancies between browsers, and different versions of the same browser
  • …and some things are not possible with it (e.g. DVD reading or exotic video formats)
  • possible concern for privacy (even if building private web tools is possible)
  • where is your data?

The web browser as universal platform?


  • Building cross-platform desktop applications is hard, with a constant need for updating (especially on Mac)
  • Available features of browsers grow constantly
  • The technology context evolves - so should the software

Maintenance burden

  • Back to the technological pharmakon: digital technology power comes with a cost
  • Who maintains the software?
    • Update something that is still working
    • Collaboration, Preservation and Sustainability in Digital Humanities: a question of time (DH2023)
    • Le Soin des Choses (Jérôme Denis, David Pontille)

Data preservation

Data Standards: requirement for data perennity, but not enough.

  • Importance of preserving data and uses
  • Standards now exist (Video in HTML, WebAnnotation)
  • Issue: update data format/model to conform new standards
    • Expert Group on Video Annotation Interoperability (VAINT) - 2020/2021 (with ELAN/VIAN/Frametrail/MediaSuite…) to determine a common model

Software preservation

  • Research output is not just the data itself
  • It is also the methodologies and practices enabled by the tools and their possibilities
  • They should be preserved too
  • Open Source/Free Software is a step towards software perennity…
  • …but it need contributors/maintainers/funding
  • At stake: build a strong, diverse and interdisciplinary community

Usability matters

  • Accessibility / ease of use
  • Aim for simplicity - KISS principle
    • text-based microsyntax (flexible, fast entry)
    • usage of timestamps (and not frame numbers/SMPTE) and simple player control
  • Co-conception dialogue / user-lead innovation
    • Instrumental genesis through interdisciplinary collaboration - reflections on the emergence of a visualization framework for video annotation data (EADH 2021)

What about AI?

  • AI is doing huge leaps
  • Building UIs for AI algorithms is not immediate
    • in Advene: data import, frontend plugins, APIs
  • Stressing the need for appropriate visualisation/interaction tools so that content can be evaluated/curated
    • HCI research, data modelling/evaluation

Summary 1/2

  • Video annotation is a complex task
    • Genericity vs. simplicity
    • Automation vs. specificity
  • Some guiding principles in Advene
    • Simplicity (text-based, micro-syntax, tags)
    • Flexible data structure and interface
    • Allow non-consistent data (but warn about them)

Summary 2/2

  • Annotations are precious
    • Use standards (WebAnnotation, IIIF…)
    • What is the quality of the data if it was mass/AI-produced?
    • How secure is your data on online platforms?
  • Maintenance is key
    • Build communities

Thank you!

Your questions and remarks are welcome