User:Brooke Vibber/Media handler and renderer thoughts
This is an essay. It expresses the opinions and ideas of some mediawiki.org users, but may not have wide support. Feel free to update this page as needed, or use the discussion page to propose major changes. |
Some thoughts on media handlers
Key problem to solution mappings
editAt config time:
- tricky to add new file types from an extension if they're not in the core mime.types
- add a registry of file extension -> mime type mapping overrides
At upload time:
- existing upload-time validation tends to be overzealous about .zip-based formats or otherwise gets confused
- add an upload-time validation function in the Handler
At transform & parser time:
- srcset is inefficiently handled in Linker
- allow handlers to do their own srcset sub-transforms
- add suitable entries to imageinfo for additional resolutions for InstantCommons
- need reliable way to attach JS modules for file display progressive enhancement
- for pre-rendered:
- have a way to transfer list of modules/styles to load from a MediaTransformOutput to a ParserOutput when adding the rendering
- for dynamic rendered:
- hang tiny stub modules in on all page views, that hook the on-wikitext-added event and check if they need to load more
- for pre-rendered:
- Rendering remote files of locally-unknown type
- add common infrastructure for iframe embedding like TMH has
- expose iframe embed URL in imageinfo response; client side can use it if no local handler available
- Render files with backing type X but display type Y
- mark photo spheres, panoramas etc with additional metadata that gets transferred across remote repos, so correct rendering gets picked
- same can handle things like rendering a PNG/TIFF source to JPEG thumbnails vs PNG depending on photo vs diagram
At MMV popup activation time:
- Rendering remote files of locally-unknown type
- Let MMV handle zoom of remote types it doesn't know locally via the iframe embedding
Background: current state of media bits
edit- mime type is primary identifier for picking a Handler class
- extension -> mime type and other -> mime type mappings are hidden and very hard to extend
- upload-time validation of file type & safety is hidden and very hard to extend
- File class encapsulates for a file from a given source repo:
- how to get the Handler instance
- wraps the Handler class's metadata & transform processes
- caches metadata via DB
- over InstantCommons, ForeignAPIFile speaks to repo over HTTP... up to once per rendering (cached)
- Handler classes encapsulate for a given file type:
- how to extract metadata from the file
- whether rendering to a thumbnail is possible
- validating thumbnail parameters from wikitext or other source
- performing some kind of backend rendering
- returning a suitable Thumbnail subclass instance with a rendering
- MediaTransformOutput classes encapsulate for a given file type:
- returning some HTML to actually show in wiki page renderings
- returning direct references for flat 2d thumbnail image
- Linker class encapsulates for all types:
- translating parameter strings into arrays to pass into the handler
- asking the Handler for thumbnail renderings at 1x, 1.5x, 2x
- combining the urls from the 1.5x and 2x renderings into the 1x MediaTransformOutput instance
- note this is hella inefficient over InstantCommons -- 3 separate imageinfo requests get run
- shoving the rendering into a pretty frame
- MMV extension encapsulates for some types:
- how to copy the thumbnail into a big pretty viewer, then ask for a high-res thumbnail and switch to it when loaded
- is based on file extension of the thumbnail, not on mime type
- ad-hoc JavaScript modules attached to particular types may handle:
- progressive enhancement of the displayed thumbnail
- TMH's replacement of bare <video>/<audio> or 'popup thumbnail' with fancy player widget
- progressive enhancement of the MMV high-res view
- 3d STL extension's in-progress plugin
- currently MMV displayable-ness and plugin mapping are based on file extension of the thumbnail, which seems fragile
- currently TMH jumps through some hoops to mark ParserOutput for later loading of the modules
- no clear way to handle adding the modules dynamically when dynamically adding source through a runtime preview, etc
- progressive enhancement of the displayed thumbnail
Ok that was kinda scary, huh?