Designing a serverless video streaming pipeline

For a recent consulting project, we were looking into how to set up a serverless video streaming pipeline, to power a new over-the-top video subscription platform they are developing (think Netflix, but for a specific target audience).

They had recently outgrown their MVP stage where they used Vimeo to host their content, and were now looking to manage the hosting and distribution on their own platform for more control over the content creation process and the user experience.

It turns out it’s actually pretty simple to set this up a completely serverless proof-of-concept on AWS. Here’s how we did it.

Project requirements

For our proof-of-concept design, we focused on these main requirements:

  • HLS format support: we wanted to provide our output in the form of standards-based M3U8 playlists files, giving the front-end team a wide range of video player libraries to choose from.
  • Variable bitrate streaming: offer at least 3 quality settings, to be able to test variable bitrate streaming based on the user’s connection speed.
  • Track outputs and conversion status in a database: we want insight in what video streams we have available for users and which ones are still processing.
  • No fixed monthly costs: we wanted to show that it’s possible to build a video processing pipeline where you only pay for usage, making sure that the company only occurs costs for this platform once it starts getting used by people that subscribe to the service.

MediaConvert

I had worked with AWS Elemental MediaConvert before to merge multiple separate MP4 clips into a single video file and knew that it supported many formats, but didn’t realise at first that it supported HLS out of the box.

The way MediaConvert works is that you build a job definition, which is a JSON file containing information about:

  • Where on Amazon S3 your input files can be found
  • Where the output files should be stored
  • In what format the output should be stored
  • How you want to transform the video or audio

You then start that job, and once its done MediaConvert will neatly place your new media files in the location you have requested.

You can even specify multiple output formats in a single job. This is very useful for supporting our variable bitrate requirement. You can choose from a list of AWS-provided output profiles (which specify among other things specify the output’s video size, frame rate, bitrate) or create your own.

S3 and CloudFront

Once converted, MediaConvert will all the parts of the video stream in an S3 bucket.

One way of making the stream available to users would be simply making the contents of that S3 bucket public, but for better performance from any place over the world and for more control and insight, it’s better to distribute the stream via CloudFront, Amazon’s global CDN.

That’s as easy as creating a new CloudFront distribution, specifying your S3 bucket as the Origin, and setting up some rules around caching and CORS.

SNS, Lambda and DynamoDB

That just leaves us with taping some stuff together to actually invoke MediaConvert and track conversion status in a database.

To keep the POC self-contained without the need for a front-end interface, we opted to create a Lambda function that is triggered whenever a file is dropped inside our S3 bucket with the prefix ‘input/’. That Lambda then prepares and fires off the MediaConvert job.

For keeping track of the conversion status, we used Amazon’s SNS (simple notification service) and asked it to listen for MediaConvert status updates. A separate Lambda function then subscribes to this SNS topic to listen for status updates, and records them together with the output URL in DynamoDB.

We used the Serverless framework to orchestrate all the different pieces of the platform, to make deployment as simple as running yarn deploy.

View the code on GitHub →

Entrepreneur tech kid, co-founder of NearSt, Londoner, open source enthusiast and aspiring spare time literature geek.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store