Real-time Correction for AI Captions and Speech

This section is part of the AI Live Streaming Manual.

Introduction

This manual explains how to use the Real-time Correction Room to correct errors in closed captions generated by AI (or human subtitling). Correctors can improve AI captions and speech in multiple ways:

errors can be corrected (e.g. names, acronyms) and punctuation can be improved
correctors can mark terms (e.g. brand names) that shouldn't be translated
the distribution of terms across lines and captions can be changed to improve readibility of the closed captions
the intelligibility of AI speech can be improved by simple interventions in sentence structure

For a first-time user, we recommend watching the video tutorial first. It will give you a impression of how things works. Note that you can use the corrector room in different ways. If you only want to correct words, you can skip the sections on shortcuts and automation.

We strongly recommend looking at the requirements and best practices before using the correction room for a real event.

Watch the video tutorial

The correction room interface

This is the interface used for correction of closed captions. It looks like this:

The interface consists of the following elements:

Connect/Disconnect button: before a user can start working, they have to connect to the correction room (in order to receive the incoming stream and send outgoing text).
Player: plays the video and floor audio as soon as it is being broadcast.
Connection status: displays the status of the incoming captions and the outgoing text.
Event status: the live event status. In order to make the room available for use, it is set to Preview (for testing in advance, viewers can not see or hear the live stream), Started (live) or Paused (viewers can not see or hear the live stream). If set to Inactive or Ended, the room is not available.
Disable/enable captions button: (temporarily) disable captions from being sent to the stream.
Captions: a separate box is shown for each closed caption. By editing the text in these boxes, you can correct the closed captions before they are shown in the video player.
Messages window: allows a user to communicate with other subtitlers for the same language and with their event managers. It also contains system messages (e.g. when someone joins or leaves the room).
Participants window: names of other users for the same language.
Keyboard shortcut list: use these custom shortcuts during correcting.
Search and replace function: replace incorrect spelling of words with the correct spelling.

Entering the correction room

As a corrector, you should receive a secure link from the event manager. Copy the link to your browser (preferably Firefox) to go to your correction room. Enter your name (for the text chat) and press 'Connect' to enter the room.

You only have access to the room if the event status is Preview, Started or Paused. If the broadcast hasn't been started, the room will look like the image below.

If the broadcast is running, you should see green check marks for both captions and text. If this isn't the case, refresh your page and try again, or use a different browser. If you still don't see both green check marks, contact your event manager.

After the broadcast is started, the video player will automatically start playing. As soon as someone speaks in the stream, the captions will appear on the right and you can start correcting them.

The text chat can be used to communicate with other correctors or the event manager(s). When other correctors and event managers connect, you will see their name in the ‘Other Participants’ window.

The correction process

As the video plays, the result of the speech-to-text conversion is displayed on the right in the form of closed captions. Each white section consists of one or two lines, which will be displayed as closed captions (and/or used as the source of AI speech).

Closed captions being generated while the broadcast running

To edit a caption, use your mouse to click on it and your keyboard to edit the text. While you're doing this, the caption will get a blue marker (on the right of it).

Color codes

Each caption has a color code indicating how much time you have left to make corrections:

Green means you still have plenty of time left.
Orange means you have some time left, but if you want to make corrections to this caption, you should make them soon.
Red means time is almost up for correcting this caption, and that it will soon be pushed to the stream.
Grey means the caption has been sent to the stream and can’t be adjusted anymore.

Using shortcuts

The correction room supports a number of shortcuts that allow correctors to do a faster and more productive job. If you plan to use these, we recommend to watch the video tutorial, where the use of each shortcut is explained more clearly.

Formatting

The correction room supports the same basic text editing techniques familiar to word processors. Double click to select a word, triple click to select a sentence. Basic visual formatting is also supported through these shortcuts:

	Mac	Windows
Bold	Ctrl + B or Cmd + B	Ctrl + B
Italic	Ctrl + I or Cmd + I	Ctrl + I
Underline	Ctrl + U	Ctrl + U
Toggle capitalisation to lower case, only first letter upper case, and upper case	N.A.	Shift + F3

These shortcuts let you move the cursor, delete words and select text:

	Mac	Windows
Move cursor to beginning of current or previous word	Option + ← (left arrow)	Ctrl + ← (left arrow)
Move cursor to end of current or next word	Option + → (right arrow)	Ctrl + → (right arrow)
Delete previous word	Ctrl + Option + Delete	Ctrl + Backspace
Delete next word	N.A.	Ctrl + Delete
Select all text	Cmd + A	Ctrl + A

While editing, you are working inside a single caption. To jump from one caption to another, use the following shortcuts:

	Mac	Windows
Next caption	Tab	Tab
Previous caption	Shift + Tab	Shift + Tab

Clear processed captions

Finally, there is a shortcut to clear all processed captions from the window, so you can resume editing at the top of the window. If you don't do this, Clevercast will do so every time the bottom of the window is reached.

	Mac	Windows
Clear processed captions	Option + O	Ctrl + O

Moving text to other captions

In some cases, it makes sense to move words from one caption to another. For example, if a caption starts with the last word of a sentence, you may want to move it to the previous caption. This is limited to a maximum of 3 words, otherwise the timing of the captions would go out-of-sync.

	Mac	Windows
Move text to previous caption (starting from cursor's location)	Option + J	Ctrl + J
Move text to next caption (starting from cursor's location)	Option + K	Ctrl + K

Marking terms as non-translatable

Some terms, like brand names, should not be translated by AI. To achieve this, the term must be added to the event's vocabulary. A corrector can do this by selecting a word or word-group and then use the following shortcut:

	Mac	Windows
Don't translate the selected word(s) (prevent the term from being translated in the additional caption languages)	Option + Y	Ctrl + Y

Once a term (with translations) is part of an event's vocabulary, that term will appear highlighted in the text. This way, the corrector knows that custom translations have already been set for this term. In that case, using the 'no-translate' shortcut for this term will no longer have effect.

Example of the term 'Clevercast' being highlighted (part of the vocabulary, with translations)

Autocorrect terms

To avoid having to make the same correction multiple times, you can update the event's vocabulary to automate a correction.

If you replace a term by another one, both terms automatically appear in the 'Autocorrect panel' in the bottom-left corner, as an 'autocorrect' suggestion. Press the 'Apply' button to add the 'Replace' and 'With' values to the vocabulary.

If a term consists of 5 characters or more, and you only change 3 characters or less, this term is automatically added to the vocabulary (instead of showing it as a suggestion). If you don't agree, you can still press the 'Undo' button.

Example

Adding 'Autocorrect' terms to the vocabulary

In the image above, two 'Autocorrect' terms are shown:

Replace translate at home by Translate@Home: this is already automated. Press the 'Undo' button if you want to cancel it.
Replace clevercast by Clevercast: this is not yet automated. Press the 'Apply' button to automate it.

Capital letters

Vocabulary terms are case-insensitive. For example, if you add Clevercast to the vocabulary, it will automate clevercast and CLEVERCAST being replaced with 'Clevercast'.

Implicitly, this means that vocabulary terms can only be capitalised in a single way. For example, once Clevercast is part of the vocabulary, it is no longer possible to add clevercast or CLEVERCAST as a term.

So if a term starts with an uppercase letter because it is at the beginning of the sentence, you must change its first letter to lowercase before adding it to the vocabulary.

If a term contains capital letters, they will always be capitalised no matter where in the sentence.
If a term starts with a lowercase letter, it will only be capitalised at the start of a sentence.

Manually adding terms

The autocorrect list contains the five most recent suggestions and corrections. You can also type terms yourself to add them to the vocabulary.

Apart from regulary autocorrection (= replacing a term with another one), there are also two other cases (for convenience):

Leaving the 'Replace' field empty: this can be used to add a term to control its capitalisation. For example, to make sure that Clevercast always gets the correct capitalisation (this is equal to typing clevercast in the 'Replace' field)
Leaving the 'With' field empty: this can be used to automatically remove certain words from the closed captions (e.g. disfluency words or profanities). This may have unintended consequences, so be very careful using this.

The vocabulary interface

Clevercast prevents you from adding the same term to the vocabulary multiple times, and from adding contradictory terms. If that happens, you will see an error message. Note that some terms may have been added to the vocabulary before the live stream starts.

The vocabulary interface provides you with an overview of all terms. In exceptional cases, you may need it to correct previous mistakes, since the correction room doesn't allow you to delete existing terms.

Requirements

Computer

We recommend a recent computer, preferably high-end or at least mid-range with a fast processor and sufficient memory (e.g. Intel i7 with 16GB RAM) and a dedicated sound card and graphic card.

Browser

The operating system doesn’t matter. It should only be able to run a recent version of a browser with WebRTC support (make sure to install the latest update).

We recommend using Firefox. Other browsers with WebRTC support, like Google Chrome should also work, but we recommend using Firefox for a number of reasons:

We use Firefox ourselves for testing. So this browser has been tested most extensively, especially after new versions or updates.
Firefox prompts the user when access to the headset is turned off. Google Chrome, for example, doesn't always do this.
Google Chrome is more commonly used for everyday browsing. There is a higher chance of add-ons being installed that could affect performance.

If for some reason Firefox isn't available, then preferably choose Google Chrome.

Fast and stable connection to Clevercast servers

We recommend using an ethernet connection if possible, since it will ensure that the connection will remain fast and stable during the entire event. If that's not possible, a good wireless connection could suffice for AI correction or subtitling.

Users need a fast and stable internet connection to the Clevercast streaming servers located in the Netherlands.

To test this, you may receive a link to the Translate@Home Connection Test page from your project manager. This way, you can test the quality of your connection in advance.

After receiving the link, open your browser (close all unnecessary other applications) and paste the test link in the address bar. The test page will ask you to enter your name, email and location and press the Start button. The browser will establish a connection with Clevercast and start sending data packets for about 30 seconds.

After 30 seconds, the test ends and you will see a popup with the result. The details of your test will also be made available to your project manager.

Sufficient bandwidth

Users should have a bandwidth of at least 5 Mbps down. If you're not sure, make sure to test beforehand or check your available bandwidth by going to https://www.speedtest.net/. Click on the Change Server link and type ‘I3D’ in the popup dialog and select ‘Rotterdam – I3D.net’. Finally, press the GO button and wait until all tests are completed.

The Download Mbps value should be higher than 5 Mbps, otherwise you may experience difficulties in watching the original video + audio stream. The PING and Upload Mbps value are less important (the Translate@Home Connection Test provides sufficient info).

Best practices

Make sure a recent Firefox (or other browser) version is installed

Do this before you start testing. Then go to the AI correction or subtitling room and check if you can connect to video and chat.

Notes:

The OpenH264 video codec (offered by Cisco Systems, Inc) should be enabled in the Firefox plug-ins. This is the case by default. If you can’t see the video in Firefox (black screen) this plug-in is probably not enabled. Try enabling it or do a fresh install of Firefox.
If you are unable to use Firefox and you have a high-end computer, you can also use Google Chrome or another browser with WebRTC support (after testing).

Restart your computer and close all other applications

To make sure your computer is stable, restart it before using it for AI correction or subtitling. When you’re ready to start, close all unnecessary applications to prevent them from using your processor, memory or internet connection. Make sure the Operating System is not allowed to start or download automatic upgrades.

Don't use multiple browser tabs

Avoid opening rooms in multiple tabs. This can result in problems with your connections or audio quality.

Test in advance

Make sure to test beforehand. If something doesn't work, contact your event manager.

Reconnect if a connection problem persists

If a problem (e.g. local network issue) occurs during the event and the connection doesn't restore itself, quickly reconnect (press the disconnect button or refresh your page and then connect again).