Build Your First Pipeline

An end-to-end walkthrough for loading the Conviva Connect Video Session Feed: choose a feed, set up delivery, understand what lands, deduplicate, compute per-day metrics, and validate against Pulse.

Updated 2026-09-25 video-ssd, pipeline, getting started, deduplicate, interval metrics

This walkthrough takes you from nothing to a validated Video Session Feed pipeline. It assumes you have read Video Session Feed: Start Here and picked a feed. Work through the steps in order.

Step 1: Choose your feed

Decide between the daily and hourly feed before anything else, because it changes how you load and aggregate the data. Use daily for the clean one-row-per-session dataset; use hourly for lower latency if you can run the deduplication step. See Choose your feed: Daily vs Hourly for the full comparison.

Step 2: Set up delivery

Conviva Connect writes SSD files to your storage on the schedule you chose. Configure the destination that fits your stack:

Choose Parquet as the file format where you can. It is columnar, compresses well, and preserves the array and record fields (such as ErrorList and SessionTags) as native nested types rather than escaped strings.

Step 3: Understand what lands

For each delivery interval, Conviva writes the data into a dated folder. Inside a folder you will find:

  • Multiple part files (the dataset is sharded for parallel writes; commonly around 30, and the exact count is configured per account).

  • A .manifest file that lists the part files for that interval. Read the manifest first and load only the files it lists, so you never pick up a partially written folder.

Load every part file in the folder as a single logical dataset for that interval. Do not assume one file equals one shard of sessions; a session lands in exactly one part file, but which one is not meaningful.

Step 4: Load and deduplicate (hourly)

If you chose the daily feed, skip to step 5: the daily feed already gives one row per session. If you chose the hourly feed, deduplicate first. A multi-hour session repeats across hourly files, once per active hour, and each row carries the session's totals up to that hour. Keep only the latest row per session before you aggregate anything across hours:

WITH ranked AS (
  SELECT
    *,
    ROW_NUMBER() OVER (
      PARTITION BY ConvivaSessionID, StartTimeUnixMs
      ORDER BY dt DESC
    ) AS rn
  FROM hourly_ssd
)
SELECT * FROM ranked WHERE rn = 1;

Step 5: Compute per-day interval metrics

The daily feed's delivered metrics are lifetime totals per session, not per-day values. A session that started before today carries its full history in today's row. To derive interval (per-day) values from these lifetime totals, subtract the prior day's totals from the current day's, joined on ConvivaSessionID:

SELECT
  L.ConvivaSessionID,
  L.PlayingTime - IFNULL(P.PlayingTime, 0) AS IntvPlayingTime,
  L.CIRR - IFNULL(P.CIRR, 0) AS IntvCIRR
FROM today L
LEFT OUTER JOIN yesterday P
  ON L.ConvivaSessionID = P.ConvivaSessionID;

An attempt belongs to the day the session started. Use StartTimeUnix to flag it, so a session that started yesterday and ended today is not counted as a new attempt today. For the worked example, the sample data, and the SQL for other interval metrics (Plays, Video Startup Failures, Video Restart Time, and more), see Calculating Interval (day) Metrics.

Step 6: Validate against Pulse

Sanity-check your pipeline before you trust it. Pick a day, a metric, and a filter (for example, playing time for one device type), compute it from your loaded SSD data, and compare it to the same view in Conviva Pulse.

Expect close but not identical numbers. The daily feed reports lifetime totals per session, while a Pulse dashboard aggregates over its chosen interval, so the two computations differ; align the windows and they converge. If a metric is far off, first confirm you deduplicated the hourly feed and that you computed interval metrics by subtracting the prior day rather than reading the lifetime totals directly. See the Lifetime vs Interval Metrics section for why the two differ.

Where to go next