Self Validation Usecases Videos

1. Start a video.
2. Let the video play for 30sec.
3. Close the video.

1. A new Conviva monitoring session should be created when the video is started and a 'Play' state should be reported when the video starts playing.
2. The Video Startup Time metric should reflect the time between pressing play and the first frame being rendered on the screen.
3. Values for Peak Bitrate, Average Bitrate and Average Frame Rate metrics should be correctly reported.
4. The session status should change to 'Ended' when the video is closed.

Note: The timeline should also show bitrate changes as the video quality ramps up/down to ensure bitrate switching is being reported correctly.

• Attempts
• Concurrent Plays
• Video Startup Time
• Peak Bitrate
• Average Bitrate
• Average Frame Rate

1. Start a video.
2. Let the video play for 15sec.
3. Without explicitly closing the first video, select another video to play so that the first video stops and the new video starts playing. Note: depending on the application design, this can be done (for example) by changing the video to a mini player and selecting a new video; or selecting a new video from a visible carousel below the player.

1. A new Conviva monitoring session should be created when the video is started and a 'Play' state should be reported when the video starts playing.
2. When the new video is started, the session status of the first video should change to 'Ended' and a new Conviva monitoring session should be created for the new video.
3. A 'Play' state should be reported in the respective session when the new video starts playing.

• Attempts
• Concurrent Plays

1. Start a video.
2. Let the video play for 15sec.
3. Use the progress bar to seek the video forward till a few seconds before the end of the video. Then let the video finish playing.
4. Let the next video in the playlist start auto-playing.

1. A new Conviva monitoring session should be created when the video is started and a 'Play' state should be reported when the video starts playing.
2. When the user seeks the video forward and the video resumes, if the user saw buffering, the timeline should show a transition into and out of 'Seek Induced Buffering' state.
3. When the first video has ended, the session status should change to 'Ended'.
4. When the next video starts playing, a new Conviva session should be created and a 'Play' state should be reported.

• Attempts
• Concurrent Plays
• Rebuffering Ratio

1. Start a video.
2. Let the video play for 15sec.
3. Pause the video for 15 secs.
4. Unpause the video.
5. Throttle the network bandwidth to trigger buffering.
6. Once the video starts buffering, remove the throttle.
7. When the video resumes, seek the video to get to the end of content.

1. A new Conviva monitoring session should be created when the video is started and a 'Play' state should be reported when the video starts playing.
2. When the video is paused and then resumed, the timeline should show a transition into and out of 'Paused' state.
3. When the user sees buffering and then content resumes, the timeline should show a transition into and out of 'Connection Induced Buffering' state.
4. When the content ends, the session status should change to 'Ended'.

• Average % Complete
• Connection Induced Rebuffering Ratio

1. Set up a blocked / unavailable video stream using Charles proxy or a similar tool.
2. Start a video.

1. A new Conviva monitoring session should be created when the video is started.
2. The player should report a fatal error in the timeline once it has exhausted all attempts to start the video.
3. The session status should change to 'VSF'.

Note: if you use multiple CDNs, it is possible that the video player switches to the other CDN. In this case, please block the other CDNs and retry this test case.

• Attempts
• Video Start Failures (VSF)

1. Start a video.
2. While the video is playing, block the video stream using Charles proxy or a similar tool.

1. A new Conviva monitoring session should be created when the video is started and a 'Play' state should be reported when the video starts playing.
2. After the video stream is blocked and the player's buffer runs out, if the user saw buffering, the timeline should show a transition to 'Connection Induced Buffering' state.
3. The player should report a fatal error in the timeline once it has exhausted all attempts to resume the video.
4. The session status should change to 'VPF'.

Note: if you use multiple CDNs, it is possible that the video player switches to the other CDN. In this case, please block the other CDNs and retry this test case.

• Attempts
• Concurrent Plays
• Video Playback Failures (VPF)

1. Start a video.
2. Close the video before the first frame is rendered on the screen.

1. A new Conviva monitoring session should be created when the video is started.
2. The session status should change to 'EBVS' when the video is closed.

• Attempts
• Exits Before Video Start (EBVS)

1. Start a video.
2. Once the video starts playing, move the application to background (video pauses).
3. After 20sec, move the application to the foreground (video resumes).

1. A new Conviva monitoring session should be created when the video is started and a 'Play' state should be reported when the video starts playing.
2. When the application is moved to background, a backgrounding event should be reported in the touchstone timeline and the player state should change to Paused.
3. When the application is moved to foreground and video resumes, a foregrounding event should be reported in the touchstone timeline and the player state should change from Paused to Playing.

Notes:

• The backgrounding and foregrounding events could be the default native events from the device, or explicitly reported using the Conviva API.
• In some devices/applications the video may continue playing in the background. In this case, a Playing state should continue to be reported in the session.

• Average % Complete

1. Start a video. Note down the Client ID of this session.
2. Close the video and reboot the device / close and relaunch the browser.
3. Start another video. Compare the Client ID of this session to the one of Step-1.

1. The Client ID in both the sessions (before and after the reboot/re-launch) should be the same.

• Unique devices

1. Open any one of the Touchstone session URLs from the tests performed above and copy the Client ID.
2. Navigate from the Pulse menu to Video Streaming Insights -> Trends.
3. Create a filter using the Client ID.
4. Select the time range during which the tests were performed.
5. Select the 'CDNs' dimension.

1. All CDN values should be valid (ex: AKAMAI, CLOUDFRONT, INHOUSE) and no UNKNOWN values should be reported.

• CDN reporting

1. Copy any one of the Touchstone session URLs from the tests performed above.
2. From the Pulse menu, navigate to Integration -> VSI Integration -> Metadata -> Metadata Validation.
3. Select the App being tested.
4. Paste the Touchstone URL and click on 'Start Metadata Validation'.

1. Automatic validation of Required and Custom metadata tags will be performed.
2. The 'Received Values' column will list all the unique values reported in the integration.
3. A 'Pass' or 'Fail' result will be assigned to each metadata tag.
4. Any tags reported in the integration but not part of the metadata strategy will be listed under 'Unexpected Metadata'.

• Required and Custom Metadata

1. Play a video with pre-roll ads.
2. Let the pre-roll ads finish playing.
3. Let the main content play for 30 secs.
4. Close the video.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. For each ad in the pre-roll a new Conviva monitoring session should be created when the respective ad starts.
3. In the main content session, for CSAI, a Play state should be reported when the main content starts playing and a Not Monitored state should be reported for the duration of the ad pod. For SSAI, a Play state should be reported when the first ad starts playing.
4. The Video Startup Time metric in case of CSAI should reflect the time between pressing play and the first frame on the main content being rendered on the screen (i.e. the ad duration should be discounted). For SSAI, the Video Startup Time metrics should reflect the time between pressing play and the first frame of the ad being rendered on the screen.
5. In the main content session, values for Peak Bitrate, Average Bitrate and Average Frame Rate metrics should be correctly reported.
6. In the main content session, a Conviva.PodStart event should be reported when the ad break starts and a Conviva.PodEnd event should be reported when the complete ad break ends.
7. The main content session status should change to 'Ended' when the video is closed.
8. In the Ad sessions, a Play state should be reported for the duration of the ad. Ad Startup Time should reflect the time it took the ad to start playing. Ad Avg Peak Bitrate and Ad Content Length should have valid values.
9. When each ad ends, the respective Ad session should show the status as 'Ended'.

Note: The timeline in the main content session should also show bitrate changes as the video quality ramps up/down to ensure bitrate switching is being reported correctly.

• Attempts
• Concurrent Plays
• Video Startup Time
• Peak Bitrate
• Average Bitrate
• Average Frame Rate
• Ad Attempts
• Ad Concurrent Plays
• Ad Impressions
• Ad Startup Time
• Ad Avg Peak Bitrate
• Ad Content Length

1. Start a video.
2. Let the video play for 15sec.
3. Without explicitly closing the first video, select another video to play so that the first video stops and the new video starts playing. Note: depending on the application design, this can be done (for example) by changing the video to a mini player and selecting a new video; or selecting a new video from a visible carousel below the player.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. If there was a pre-roll, for each ad in the pre-roll a new Conviva monitoring session should be created when the respective ad starts. When each ad ends, their respective ad session status should change to 'Ended'.
3. After 15sec of main content playback, when a new video is started, the main content session status for the first video should change to 'Ended'.
4. A new Conviva monitoring session for the main content of the second video should be created.
5. If there was a pre-roll in the second video, for each ad in the pre-roll a new Conviva monitoring session should be created when the respective ad starts.

• Attempts
• Concurrent Plays
• Ad Attempts
• Ad Concurrent Plays
• Ad Impressions

1. Start a video.
2. After any pre-roll ads, let the main video play for 15sec.
3. Use the progress bar to seek the video forward till a few seconds before the end of the video. Then let the video finish playing.
4. Let the next video in the playlist start auto-playing.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. If there was a pre-roll, for each ad in the pre-roll a new Conviva monitoring session should be created when the respective ad starts. When each ad ends, their respective ad session status should change to 'Ended'.
3. When the user seeks the main video forward and the video resumes, if the user saw buffering, the timeline should show a transition into and out of 'Seek Induced Buffering' state.
4. When the first video has ended, the session status should change to 'Ended'.
5. When the next video starts, a new Conviva monitoring session for the main content should be created.
6. If there was a pre-roll in the next video, for each ad in the pre-roll a new Conviva monitoring session should be created when the respective ad starts.

• Attempts
• Concurrent Plays
• Rebuffering Ratio
• Ad Attempts
• Ad Concurrent Plays
• Ad Impressions

1. Start a video.
2. After any pre-roll ads, let the main video play for 15sec.
3. Pause the video for 15 secs.
4. Unpause the video.
5. Throttle the network bandwidth to trigger buffering.
6. Once the video starts buffering, remove the throttle.
7. When the video resumes, seek the video to get to the end of content.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. If there was a pre-roll, for each ad in the pre-roll a new Conviva monitoring session should be created when the respective ad starts. When each ad ends, their respective ad session status should change to 'Ended'.
3. When the main video is paused and then resumed, the timeline should show a transition into and out of 'Paused' state.
4. When the user sees buffering and then content resumes, the timeline should show a transition into and out of 'Connection Induced Buffering' state.
5. When the content ends, the session status should change to 'Ended'.

• Average % Complete
• Connection Induced Rebuffering Ratio

1. Start a video.
2. During the pre-roll ad, throttle the stream to create buffering in the ad.
3. Unthrottle when the ad starts buffering to allow the ad to finish playing.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. For each ad in the pre-roll a new Conviva monitoring session should be created when the respective ad starts.
3. When the ad is playing, there should be a 'Playing' state reported in the ad session. When the ad is buffering, the timeline should show a 'Connection Induced Buffering' state. When the ad finishes, the ad session status should be 'Ended'.
4. For CSAI, in the main content session, the entire duration of the ad break should be reported as a 'Not Monitored' state. For SSAI, a 'Playing' state should be reported when the ad is playing and a 'Connection Induced Buffering' state should be reported when the ad is buffering.

• Ad % Complete
• Ad Rebuffering Ratio

1. Set up a blocked / unavailable video stream using Charles proxy or a similar tool.
2. Start a video.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. If there was a CSAI pre-roll, for each ad in the pre-roll a new Conviva monitoring session will be created when the respective ad starts. This is expected as the ad stream URL is different from the (blocked) content stream URL. In case of SSAI, ad sessions may not be created because the ad and content stream is the same and the ad beacons only fire when the player reaches specific timestamps in the video where an ad is supposed to be. If you block the URL, the player never starts and never hits those timestamps.
3. The player should report a fatal error in the content session timeline once it has exhausted all attempts to start the video.
4. The content session status should change to 'VSF'.

Note: if you use multiple CDNs, it is possible that the video player switches to the other CDN. In this case, please block the other CDNs and retry this test case.

• Attempts
• Video Start Failures (VSF)

1. Block the ad stream using Charles proxy or a similar tool.
2. Start a video.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. For CSAI pre-roll, for each ad in the pre-roll a new Conviva monitoring session will be created. Each ad will fail to play and the ad session status will be 'ASF'. In case of SSAI, ad sessions may not be created because the ad and content stream is the same and the ad beacons only fire when the player reaches specific timestamps in the video where an ad is supposed to be. If you block the URL, the player never starts and never hits those timestamps.
3. In the main content session, for CSAI, a 'Not Monitored' state will be reported during the failed ad break and a 'Playing' state will be reported when the main content starts playing. For SSAI, since the ad and content stream are the same, the main content will also fail to start and the session will end as a 'VSF'.

• Ad Attempts
• Ad Start Failures

1. Start a video and let the main content start playing (after any pre-roll ads).
2. While the content is playing, block the video stream using Charles proxy or a similar tool.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. If there was a pre-roll, for each ad in the pre-roll a new Conviva monitoring session should be created when the respective ad starts. When each ad ends, their respective ad session status should change to 'Ended'.
3. After the video stream is blocked and the player's buffer runs out, if the user saw buffering, the content session timeline should show a transition to 'Connection Induced Buffering' state.
4. The player should report a fatal error in the content session timeline once it has exhausted all attempts to resume the video.
5. The content session status should change to 'VPF'.

Note: if you use multiple CDNs, it is possible that the video player switches to the other CDN. In this case, please block the other CDNs and retry this test case.

• Attempts
• Video Playback Failures (VPF)

1. Start a video.
2. Block the ad stream using Charles proxy or a similar tool as soon as the ad starts playing.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. For CSAI/SSAI pre-roll, for each ad in the pre-roll a new Conviva monitoring session will be created. A 'Playing' state will be reported in the ad session when the ad is playing. When the blocked ad fails to continue, the ad session status should end as an 'APF'.
3. In the main content session, for CSAI, a 'Not Monitored' state will be reported for the ad duration and a 'Playing' state will be reported when the main content starts playing. For SSAI, since the ad and content stream are the same, a 'Playing' state will be reported when the ad is playing. When the ad fails, the content session will end as a 'VPF'.

• Ad Attempts
• Ad Playback Failure

1. Start a video.
2. Close the video before the first frame is rendered on the screen. Note: use a video where there is no pre-roll, or resume a previously started video so that it starts without a pre-roll.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. The main content session status should change to 'EBVS' when the video is closed.

• Attempts
• Exits Before Video Start (EBVS)

1. Start a video.
2. Before the first frame of the pre-roll is rendered, close the video.

1. A new Conviva monitoring session for the main content and for the first pre-roll ad should be created when the video is started.
2. When the video is closed, the content session should end as 'EBVS' and the CSAI ad session should end as 'EBAS'.

Note: In case of SSAI, ad sessions may not be created because the ad and content stream is the same and the ad beacons only fire when the player reaches specific timestamps in the video where an ad is supposed to be. When we close the video before it starts, the player never hits those timestamps.

• Ad Attempts
• Exit Before Ad Start

1. Start a video.
2. Once the main content starts playing, move the application to background (video pauses).
3. After 20sec, move the application to the foreground (video resumes).

1. A new Conviva monitoring session should be created when the video is started and a 'Play' state should be reported when the video starts playing.
2. When the application is moved to background, a backgrounding event should be reported in the touchstone timeline and the player state should change to Paused.
3. When the application is moved to foreground and video resumes, a foregrounding event should be reported in the touchstone timeline and the player state should change from Paused to Playing.

Notes:

• The backgrounding and foregrounding events could be the default native events from the device, or explicitly reported using the Conviva API.
• In some devices/applications the video may continue playing in the background. In this case, a Playing state should continue to be reported in the content session.

• Average % Complete

1. Start a video.
2. During the pre-roll or mid-roll ad playback, move the application to background (ad pauses).
3. After 20sec, move the application to the foreground (ad resumes).

1. A new Conviva monitoring session should be created for the main content and separate ad sessions should be created when each ad in the ad break starts playing.
2. When the application is moved to background, a backgrounding event should be reported in the touchstone timeline of the main content session. In this content session, for CSAI, the player state should remain in a Not Monitored state. For SSAI, the player state should change to Paused.
3. In the specific ad session (during which backgrounding was done), the player state should change from Playing to Paused.
4. When the application is moved to foreground and the ad resumes, a foregrounding event should be reported in the touchstone timeline of the main content session. In this content session, for CSAI, the player state should continue in Not Monitored state till the ad ends. For SSAI, the player state should change to Playing.
5. In the specific ad session (during which foregrounding was done), the player state should change from Paused to Playing.

Notes:

• The backgrounding and foregrounding events could be the default native events from the device, or explicitly reported using the Conviva API.
• In some devices/applications the video may continue playing in the background. In this case, a Playing state should continue to be reported in the ad session (for both CSAI and SSAI ads) and content session (for SSAI ads).

• Ad % Complete

1. Play a video with mid-roll ads.

1. A new Conviva monitoring session for the main content should be created when the video is started.
2. If there was a pre-roll, for each ad in the pre-roll a new Conviva monitoring session should be created when the respective ad starts.
3. When the player reaches the mid-roll, for each ad in the mid-roll a new Conviva monitoring session should be created when the respective ad starts.
4. During the ad break, for CSAI, a Not Monitored state should be reported in the main content session. For SSAI, a Play state should be reported in the main content session.
5. In the main content session, a Conviva.PodStart event should be reported when the ad break starts and a Conviva.PodEnd event should be reported when the complete ad break ends.
6. In the ad sessions, a Play state should be reported for the duration of the ad. Ad Avg Peak Bitrate and Ad Content Length should have valid values.
7. When each ad ends, the respective Ad session should show the status as 'Ended'.

• Average % Complete
• Ad Attempts
• Ad Impression
• Ad Avg Peak Bitrate
• Ad Content Length

1. Start a video. Note down the Client ID of this session.
2. Close the video and reboot the device / close and relaunch the browser.
3. Start another video. Compare the Client ID of this session to the one of Step-1.

1. The Client ID in both the sessions (before and after the reboot/re-launch) should be the same.

• Unique devices

1. Open any one of the Touchstone session URLs from the tests performed above and copy the Client ID.
2. Navigate from the Pulse menu to Video Streaming Insights -> Trends.
3. Create a filter using the Client ID.
4. Select the time range during which the tests were performed.
5. Select the 'CDNs' dimension.

1. All CDN values should be valid (ex: AKAMAI, CLOUDFRONT, INHOUSE) and no UNKNOWN values should be reported.

• CDN reporting

1. Copy any one of the Touchstone session URLs from the tests performed above.
2. From the Pulse menu, navigate to Integration -> VSI Integration -> Metadata -> Metadata Validation.
3. Select the App being tested.
4. Paste the Touchstone URL and click on 'Start Metadata Validation'.

1. Automatic validation of Required and Custom metadata tags will be performed.
2. The 'Received Values' column will list all the unique values reported in the integration.
3. A 'Pass' or 'Fail' result will be assigned to each metadata tag.
4. Any tags reported in the integration but not part of the metadata strategy will be listed under 'Unexpected Metadata'.

• Required and Custom Metadata