Defect detectors (where each of them also runs as a separate microservice) then download images and processed it concurrently using AWS Lambda. To reduce computationally expensive video conversion jobs, we built a microservice that splits videos into frames and temporarily uploads images to an Amazon Simple Storage Service (Amazon S3) bucket. The second cost problem we discovered was about the way we were passing video frames (images) around different components. Besides that, AWS Step Functions charges users per state transition. Our service performed multiple state transitions for every second of the stream, so we quickly reached account limits. The main scaling bottleneck in the architecture was the orchestration management that was implemented using AWS Step Functions. ![]() The initial architecture of our defect detection system. The following diagram shows the serverless architecture of our service. Also, the overall cost of all the building blocks was too high to accept the solution at a large scale. However, the way we used some components caused us to hit a hard scaling limit at around 5% of the expected load. In theory, this would allow us to scale each service component independently. We designed our initial solution as a distributed system using serverless components (for example, AWS Step Functions or AWS Lambda), which was a good choice for building the service quickly. The third component provides orchestration that controls the flow in the service. For more information about this topic, see our How Prime Video uses machine learning to ensure video quality article. ![]() Defect detectors execute algorithms that analyze frames and audio buffers in real-time looking for defects (such as video freeze, block corruption, or audio/video synchronization problems) and send real-time notifications whenever a defect is found. The media converter converts input audio/video streams to frames or decrypted audio buffers that are sent to detectors. Our service consists of three major components. Because we compiled all the operations into a single process, we could rely on scalable Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Elastic Container Service (Amazon ECS) instances for the deployment. To address this, we moved all components into a single process to keep the data transfer within the process memory, which also simplified the orchestration logic. The two most expensive operations in terms of cost were the orchestration workflow and when data passed between distributed components. ![]() The initial version of our service consisted of distributed components that were orchestrated by AWS Step Functions. So, we took a step back and revisited the architecture of the existing service, focusing on the cost and scaling bottlenecks. We also noticed scaling bottlenecks that prevented us from monitoring thousands of streams. While onboarding more streams to the service, we noticed that running the infrastructure at a high scale was very expensive. Our Video Quality Analysis (VQA) team at Prime Video already owned a tool for audio/video quality inspection, but we never intended nor designed it to run at high scale (our target was to monitor thousands of concurrent streams and grow that number over time). ![]() This tool allows us to automatically identify perceptual quality issues (for example, block corruption or audio/video sync problems) and trigger a process to fix them. To ensure that customers seamlessly receive content, Prime Video set up a tool to monitor every stream viewed by customers. At Prime Video, we offer thousands of live streams to our customers.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |