CC-BY

Using Advene to accompany research in AudioVisual Digital Humanities

Sorry, your browser does not support SVG.

Olivier Aubert - www.olivieraubert.net
FOSDEM 2020 - 01/02/2020

Plan

  • The Advene project
  • Use cases in Digital Humanities
    • REMIND method
    • AdA project
  • Conclusion

Advene project

Annotate Digital Video, Exchange on the NEt

to be added to something or become a part of it, though inessential (Webster, 1913)

  • Accompany active reading of AV documents through annotations
  • Create, user, share analyses of AV documents as hypervideos
  • Started in 2002 with Yannick Prié and P.A. Champin
  • Free Software (GPL) cross-platform desktop application (Python/Gtk/Gstreamer)
  • www.advene.org

Interface

screenshot_advene.png

Principle

Sorry, your browser does not support SVG.

Scientific challenges

  • Knowledge engineering
  • Document engineering
  • HCI
  • Dataviz
  • Activity traces

Features

  • User defined structures and visualizations (templates, rules…)
  • Preserve structured annotations as long as possible in the workflow
  • A package is a set of (annotations, annotation structure, views, queries) independent from the video
  • Generic data model. Use of text-based microsyntax.
    • import filters
    • no strict constraints
  • Flexible and extensible (through templates and plugins)
  • Offline work

2 use cases in Digital Humanities

  • Interview analysis in museology : REMIND
  • Systematic quantitative analysis and semantic annotation in media studies : AdA project

REMIND method

Goal: study museum visitors' perception through video-based autoconfrontation

remind.jpg

  • Work by Daniel Schmitt (Univ. Valenciennes)
  • Self-explicitation of visitors through replaying subjective recording of the visit
  • Online publication of produced visualizations and raw data

Example: Museographie

Analysis process

  • Transcribe interview (note-taking view)
  • Identify hexadic signs (from course-of-action methodology)
  • Use relations to express courses of experience
  • Generate visualizations through templates
  • Write analyzes
  • Publish on the web

REMIND method

Exploratory structuration / Publication of analyses as hypervideos

www.museographie.fr

screenshot_museographie.png

AdA project

www.ada.cinepoetics.fu-berlin.de

  • Affektrhetoriken des Audiovisuellen - Audio-visual rhetorics of affect
  • Systematic-empirical study of staging patterns in audiovisual representations of the 2007 financial crisis
  • Interdisciplinary collaboration: Cinepoietics (Media Studies, Freie Universität Berlin), HPI Potsdam (semantic web, video analysis)

Challenges and needs

Challenges

  • Developing a methodological framework for analysis, with quantitative support
  • Annotation content semantization
  • Exploration of automation possibilities

Needs

  • Build a ground truth for future automation
  • Optimize the manual annotation process
  • Provide a bridge between users and semantic data

Application

advene_ada.png

Process

  • Exploratory modelling of an ontology through Advene schemas definition
  • Conversion of initial Advene schemas into bootstrap ontology
  • Conversion of OWL ontology into Advene schema
    • convert properties (URIs) into "local" vocabularies (folksonomies)
    • preserve URI/vocabulary mapping through metadata
  • Non-blocking data validation feedback
  • Export of information into an extended WebAnnotation model

Principle

Sorry, your browser does not support SVG.

Produced ontology (HPI)

ada.filmontology.org

  • 8 annotation levels: Segmentation, Camera, Montage, Image Composition, Acoustics, Language, Bodily expressivity, Motifs
  • 78 annotation types
  • (currently) 86000 annotations over 5 movies and many TV reports
  • CC-SA 3.0

Data visualization

visualization

ada_timeline.png

Advene developments for the AdA project 1/2

HCI

  • Update code for adapting to new systems (esp. Python3/Gtk3/Gstreamer1)
  • Performance improvements
  • Constraint vocabularies support
  • Data visualization

Advene developments for the AdA project 2/2

  • Improvements of fusion/import features for collaboration
  • Semantic web interoperability
    • Ontology import
    • RDF export (WebAnnotation + extensions)
  • Integration of a concept-detection server from HPI

Conclusion

  • Advene - a useful tool for AV Digital Humanities project
    • Able to export/import from multiple other models/formats
    • Flexible and extensible
    • Supports the whole scholar workflow from exploration to publication
  • Available support for custom developments and training - contact me
  • Many ideas, looking for funding

Thanks for you attention

Any question?

@Olivier_Aubert
www.olivieraubert.net
www.advene.org

Appendix: Active Reading

Sorry, your browser does not support SVG.