This prototype makes use of AWS Interactive Video Service, giving us the ability to send a timed metadata event at any time while streaming and guarantee the event is received at the same point in playback by all viewers regardless of latency. This allows for near-realtime interactive video experiences to be created with confidence that time-dependent components are presented at the correct moment and users won’t become confused by dynamic content rendering out of turn. Several prototype features are built around this technology.
Gathering contemporaneous audience feedback is a common streaming tactic. An admin/host can create a poll and its choices in the admin app and send an activate-poll event with the poll’s data; the client app will listen for the event and render an interactive poll element when that event has been received. A user can select from the poll’s list of choices and submit their response, or minimize the poll element if they want to focus on chat. The admin/host can either deactivate the poll, which sends an event that upon receipt will dismiss the poll element, or choose to reveal the results, which sends an event containing each choice’s response count and is rendered inline with the poll options.
Text prompts are created in a similar manner to Polls. The admin/host creates a text prompt and sends a metadata event which upon receipt by the client renders a minimal form input. Participants respond to text prompts by typing their response and submitting the form. Responses are processed differently, however — each response is sent to a Lambda function running a simple topic extraction algorithm: using Spacy, reduce each response to a set of nouns, proper nouns, and adjectives and ignore everything else. The set of keywords is stored along with the raw response. Common keywords for each text prompt are bucketed and displayed to the admin/host, who can then use the distribution of keywords in the live stream as they see fit. Given that a host/admin might be simultaneously fulfilling other responsibilities, this approach offers several advantages over attempting to understand what users are saying by visually parsing a potentially very noisy chat log:
intentionality on the broadcaster’s part in the feedback they solicit
visibility for users in that text prompts are distinct from the normal chat flow, and can be styled a number of ways
host/admin doesn’t have to interrupt their workflow to sift through chat messages
We created a utility app for the broadcaster to manage interactive elements, monitor chat, and send metadata events. Users of the admin app can manage polls and prompts (standard CRUD actions and sending relevant timed metadata events), chat with users, and see more detailed information while also keeping track of the video stream in a compact view.
Logged in users and the host/administrator can converse in realtime with Websockets-based chat. System-level messages (such as when a user joins) are also surfaced here. Chat could be handled with IVS’s Stream Chat API, which at the time we built this feature was not available.
These features represent fairly common use cases that PBS and its member stations would encounter when streaming to engage an audience, but there are many ways to put this technology to work. An early prototype demonstrated the use of HTML Canvas to dramatically animate text and images overlaid on the video stream when a timed metadata event fired.
Anything you might want to show to viewers reliably synchronized with video is fair play. Think donation solicitations, or surfacing a link to a partner or sponsor or promoted PBS program, or a modern “telethon” with a leaderboard that updates as pledges roll in. Think of tying an incoming timed metadata event to a switch on a physical component, like a dunk tank or a rail switch on a model train set. We encourage all ideas and would love to explore the possibilities with streamers like you.
