Advene, 20 years of video annotation

Olivier Aubert - www.olivieraubert.net
LS2N - Nantes Université

Reimagining Annotation for Multimodal Cultural Heritage - 08/02/2024

Outline

• Context: annotations
• Advene - principles and experiments
• Lessons learned

Annotations in research

Annotation - one of the fundations of scholarship work

Page of the Codex Oxoniensis Clarkianus 39 (Clarke Plato). Dialogue Gorgias. Public Domain

More recent annotations

CC Public Domain

Now digital…

Various video annotation communities

Ecology of hypertext annotation

Advene project

Annotate Digital Video, Exchange on the NEt

to be added to something or become a part of it, though inessential (Webster, 1913)

Create, use, exchange analyses of audiovisual documents as hypervideos

• Started 2001 with Yannick Prié and P.A. Champin
• Free software (GPL) cross-platform desktop app.

www.advene.org

Advene

Scientific motivation

• Knowledge engineering
• Document engineering
• Human-Computer Interfaces
• Datavisualization
• Activity traces

Scratch an itch…

Mulholland Drive (2001) by David Lynch

Advene focus

• Active reading / close reading
• Single video
• Companion to exploratory research
• Flexibility (structures, contents, outputs…)
• Offline use

Principle

Different levels of flexibility

to accompany the exploratory analysis

• Variety of supported video formats (video file, DVD, stream)
• Flexibility in annotation structure
  • Dynamic (re)structuring
  • Allow non-conform data (but warn)
• Variety of annotation creation interfaces
  • Manual annotation with possible assistants
  • Using automatic results as base
• Flexibility in annotation visualization
• Extensibility through plugins

Projects timeline

• Incorporactions (2004-2007)
• Cinelab (2007-2008)
• CineCast (2009-2011)
• datadata (2010-2011)
• ACAV (2009-2012)
• Il Caso Mattei (2010-2011)
• Museographie (2011-2013)
• SpectacleEnLignes (2011-2015)
• (CocoNotes) (2013-2016)
• AdA project (2017-2021)

Usage

• Lab experiment analysis (SVG annotations)
• Accessibility for blind users (Braille display)
• Movie analysis : Il caso Mattei (graphs), AdA (input/ontologies)
• Interview analysis : Museographie (structure/hypervideo)

Example: Lab experiment analysis

A Case Study of Object and Occlusion Management on the eLabBench, a Mixed Physical/Digital Tabletop (Aurélien Tabard - University Lyon 1/CNRS)

• Graphical annotation of video capture of a laboratory bench
• Study of user behaviour through most used bench areas

Example: Lab experiment analysis

Lab experiment - Example output

Quantitative processing of graphical annotations

Example: accessibility

Dynamic annotation rendering through a Braille display

Example: Il caso Mattei

Work with Livia Giunti for her PhD about digital tools and time representation in movies

• Importance of digital analysis tools
• Comparison of different tools
• Study of annotation structuring
• Definition of visualisations dedicated to temporality (shot duration bargraph, chronologic vs diegetic time)

Example: Il caso Mattei

Dedicated datavisualization

Example: Museographie

Objective: study visitor's perception of museum exhibits through analysis of visitor video interviews

Example: Museographie

Analysis process

• Transcribe interview (note-taking view)
• Identify hexadic signs (from course-of-action methodology)
• Use relations to express courses of experience
• Generate visualizations through templates
• Write analyzes
• Publish on the web

Example: Museographie

Exploratory structuring
Analysis publication as custom hypervideo

Example: AdA project

https://projectada.github.io/

• Affektrhetoriken des Audiovisuellen - Audio-visual rhetorics of affect
• Systematic-empirical study of staging patterns in audiovisual representations of the 2007 financial crisis
• Interdisciplinary collaboration with Cinepoietics (Media Studies, Freie Universität Berlin) and HPI Potsdam (semantic web, video analysis)

AdA project - Challenges and needs

Challenges

• Developing a methodological framework for movie analysis, with quantitative support
• Semantic video-annotations
• Exploration of automation possibilities (feature extraction)

Needs

• Provide a 2-way bridge between users and semantic data
• Optimize the manual annotation process
• Build a ground truth for future automation

AdA project - Advene screenshot

AdA project - Principle

ADA project - Developments

• maintain code to support recent libraries (Python3 / Gtk3)
• improve performance
• interface developments for constrained vocabularies
• improve merging/importing features for collaboration
• bridge for semantic web interoperability (ontology import, rdf export)
• import from HPI concept detection server
• additional visualisations (web timeline)

Lessons

• The technology pharmakon
• The maintenance burden
• Usability matters

Technology pharmakon

• Technology is a requirement in Digital Humanities - epistemic change
• Technology is moving and fragile
• Technical debt can accumulate

Video and technology

• Video material does not exist without technology
• Initial design decision to use DVD as main source of shared, legit video
  • Simple player abstraction, handling mainly time dimension
• In 2001: video not first class citizen on the web
  • <video> tag - initial draft in 2007, official W3C recommandation in 2014
  • in web context, use of other tools (flash - deprecated in 2017)
• What about 360 video? VR experiences?

Annotation standards

• Many initiatives in 2001 (Mosaic, Annotea…)
• WebAnnotation W3C working group started in 2014 (after OpenAnnotation)
• WebAnnotation standardized in 2017
• For preceding models, model adaptation may be needed

The web browser as universal platform?

• The browser has become the de-facto universal runtime…
• …but beware of small discrepancies between browsers, and different versions of the same browser
• …and some things are not possible with it (e.g. DVD reading or exotic video formats)
• possible concern for privacy (even if building private web tools is possible)
• where is your data?

The web browser as universal platform?

But

• Building cross-platform desktop applications is hard, with a constant need for updating (especially on Mac)
• Available features of browsers grow constantly
• The technology context evolves - so should the software

Maintenance burden

• Back to the technological pharmakon: digital technology power comes with a cost
• Who maintains the software?
  • Update something that is still working
  • Collaboration, Preservation and Sustainability in Digital Humanities: a question of time (DH2023)
  • Le Soin des Choses (Jérôme Denis, David Pontille)

Data preservation

Data Standards: requirement for data perennity, but not enough.

• Importance of preserving data and uses
• Standards now exist (Video in HTML, WebAnnotation)
• Issue: update data format/model to conform new standards
  • Expert Group on Video Annotation Interoperability (VAINT) - 2020/2021 (with ELAN/VIAN/Frametrail/MediaSuite…) to determine a common model

Software preservation

• Research output is not just the data itself
• It is also the methodologies and practices enabled by the tools and their possibilities
• They should be preserved too
• Open Source/Free Software is a step towards software perennity…
• …but it need contributors/maintainers/funding
• At stake: build a strong, diverse and interdisciplinary community

Usability matters

• Accessibility / ease of use
• Aim for simplicity - KISS principle
  • text-based microsyntax (flexible, fast entry)
  • usage of timestamps (and not frame numbers/SMPTE) and simple player control
• Co-conception dialogue / user-lead innovation
  • Instrumental genesis through interdisciplinary collaboration - reflections on the emergence of a visualization framework for video annotation data (EADH 2021)

What about AI?

• AI is doing huge leaps
• Building UIs for AI algorithms is not immediate
  • in Advene: data import, frontend plugins, APIs
• Stressing the need for appropriate visualisation/interaction tools so that content can be evaluated/curated
  • HCI research, data modelling/evaluation

Summary 1/2

• Video annotation is a complex task
  • Genericity vs. simplicity
  • Automation vs. specificity
• Some guiding principles in Advene
  • Simplicity (text-based, micro-syntax, tags)
  • Flexible data structure and interface
  • Allow non-consistent data (but warn about them)

Summary 2/2

• Annotations are precious
  • Use standards (WebAnnotation, IIIF…)
  • What is the quality of the data if it was mass/AI-produced?
  • How secure is your data on online platforms?
• Maintenance is key
  • Build communities

Thank you!

Your questions and remarks are welcome

@Olivier_Aubert@piaille.fr
www.olivieraubert.net
www.advene.org