Play Video File Backwards

In this tutorial, you will learn how to play a video file backwards efficiently and smoothly with OpenCV C++ functions using multiple threads.

Note : To understand this tutorial better, please refer to  how to play a video file forwards first.


When a video file is played backwards, last frames should be decoded first. Therefore you have to call VideoCapture::set(CAP_PROP_POS_FRAMES, index) function to jump to last frames of the video before calling the VideoCapture::read(frame) function.

But this VideoCapture::set(CAP_PROP_POS_FRAMES, index)  function call is expensive in terms of the CPU usage. Therefore this function should not be called inside the while loop in the main thread because it will slow down the video playback. Instead, a separate thread (We will call this thread, capturing thread) should be spawned to decode frames from the video file by calling VideoCapture::set(CAP_PROP_POS_FRAMES, index) function and VideoCapture::read(frame) function. The while loop in the main thread should only display the video frames decoded in the capturing thread.

But if you call VideoCapture::set(CAP_PROP_POS_FRAMES, index) function in each iteration of the loop in the capturing thread, capturing thread will slow down and not be able to supply frames to main thread at the rate required for the video playback. Therefore you have to minimize the number of calls to the VideoCapture::set(CAP_PROP_POS_FRAMES, index) function as much as possible in the capturing thread. This can be achieved by buffering certain number of frames before transferring them to the main thread. The buffer size should be decided considering the available memory in your computer, because decoded video frames significantly consume your memory.

This is how you should decode frames in the capturing thread. Following assumptions are made to simplify the logic.
  • Total number of frames in the video is 13
  • The buffer size is 5.


  • VideoCapture cap(location_of_video_file); 

  •  cap.set(CAP_PROP_POS_FRAMES, 8);
  • cap.read(frame); //decoding the frame at 8
  • cap.read(frame); //decoding the frame at 9
  • cap.read(frame); //decoding the frame at 10
  • cap.read(frame); //decoding the frame at 11
  • cap.read(frame); //decoding the frame at 12
  • Add the 5 frames in the local buffer to the shared buffer  

  • cap.set(CAP_PROP_POS_FRAMES, 3);
  • cap.read(frame); //decoding the frame at 3
  • cap.read(frame); //decoding the frame at 4
  • cap.read(frame); //decoding the frame at 5
  • cap.read(frame); //decoding the frame at 6
  • cap.read(frame); //decoding the frame at 7
  • Add the 5 frames in the local buffer to the shared buffer

  • cap.set(CAP_PROP_POS_FRAMES, 0);
  • cap.read(frame); //decoding the frame at 0
  • cap.read(frame); //decoding the frame at 1
  • cap.read(frame); //decoding the frame at 2
  • Add the 3 frames in the local buffer to the shared buffer


It should be noted that VideoCapture::set(CAP_PROP_POS_FRAMES, index) function is called only 3 times to decode 13 video frames at the expense of buffering 5 video frames in the memory.

Higher the buffer size, the number of calls to the VideoCapture::set(CAP_PROP_POS_FRAMES, index) function will decrease. But it will increase the memory consumption of your program. Therefore you have to decide the buffer size carefully in your program.


This is how you display the extracted frames in the video in the main thread.


  • Move all  the available frames in the shared buffer to the local buffer
  • Display the frame at 12 and remove it from from the local buffer
  • Display the frame at 11 and remove it from from the local buffer
  • Display the frame at 10 and remove it from from the local buffer
  • ...
  • ...
  • ...  

  •  When the local buffer is empty, move all the available frames in the shared buffer to the local buffer
  • Display the frame at (x) and remove it from from the local buffer
  • Display the frame at (x - 1) and remove it from from the local buffer
  • Display the frame at (x - 2) and remove it from from the local buffer
  • ...
  • ...  

  •  When the local buffer is empty, move all the available frames in the shared buffer to the local buffer

  • ...
  • ...

  • Display the frame at 2 and remove it from from the local buffer
  • Display the frame at 1 and remove it from from the local buffer
  • Display the frame at 0 and remove it from from the local buffer



This is the implementation of the above logic with OpenCV C++ functions.

//Uncomment the following line if you are compiling this code in Visual Studio
//#include "stdafx.h"

#include <opencv2/opencv.hpp>
#include <iostream>
#include <thread>   
#include <mutex> 
#include <atomic>

using namespace cv;
using namespace std;




//Mutex used to protect the shared_frame_buffer variable
mutex mtx;

//The capturing thread will insert video frames to this list
//Main thread will take out video frames from this list
list<Mat> shared_frame_buffer; 

//This boolean will be set to true after the capturing thread finishes its execution
atomic<bool> capturing_thread_finished = false; 

//This boolean will be set to true after the main thread finishes its execution
atomic<bool> main_thread_finished = false; 

//The time interval in milliseconds between consecutive frames to achieve the specified frame rate
atomic<int> frame_interval = -1; 

//Size of the list in the main thread which contains frames taken from the shared_frame_buffer
atomic<int> main_thread_buffer_size = 0; 




void capture_frames()
{
 //open the video file for reading
 VideoCapture cap("D:/My OpenCV Website/A Herd of Deer Running.mp4");

 // if not success, exit program
 if (cap.isOpened() == false)
 {
  cout << "Cannot open the video file" << endl;
  capturing_thread_finished = true;
  return;
 }

 //get the time interval of consecutive frames in milliseconds to achieve the frame rate of the video 
 frame_interval = static_cast<int>(1000 / cap.get(CAP_PROP_FPS));

 //get the total frame count of the video
 int remaining_frame_count = static_cast<int>(cap.get(CAP_PROP_FRAME_COUNT));
 
 //sleeping time in microseconds
 int sleeping_time = 1;
 int sleeping_time_increment = 100;

 //High buffer size - Memory consumption will increase, CPU consumption will decrease
 //Low buffer size - Memory consumption will decrease, CPU consumption will increase
 int buffer_size = 20;

 while (true)
 {
  //If the main thread has finished its execution, there is no point of continuing this loop
  if (main_thread_finished)
   break;

  list<Mat> local_frame_buffer;

  //Finding a starting frame number to start decoding the frames in the video
  int starting_position = remaining_frame_count - buffer_size;
  
  if (starting_position < 0)
   starting_position = 0;

  //Setting the starting frame number to be decoded next in the video. 
  //This function call is expensive in terms of CPU consumption.
  cap.set(CAP_PROP_POS_FRAMES, starting_position);

  for (int i = starting_position; i < remaining_frame_count; ++i)
  {
   Mat frame;
   bool bSuccess = cap.read(frame); // read a new frame from video 

   assert(bSuccess); //bSuccess variable should be always true

   local_frame_buffer.push_front(frame);

   //This line sleeps for a time duration in microseconds as specified in the sleeping_time variable.
   this_thread::sleep_for(chrono::microseconds(sleeping_time));
  }

  remaining_frame_count = starting_position;

  mtx.lock();
  shared_frame_buffer.splice(shared_frame_buffer.end(), local_frame_buffer);
  int total_frame_buffer_size = static_cast<int>(shared_frame_buffer.size()) + main_thread_buffer_size;
  mtx.unlock();

  //This thread has finished decoding frames of the video
  if (starting_position == 0)
   break;
  
  //This section will control the speed of execution of this thread
  if (total_frame_buffer_size > buffer_size * 3)
  {
   sleeping_time = sleeping_time + sleeping_time_increment;
   sleeping_time_increment = min(5000, sleeping_time_increment * 2);
  }
  else
  {
   sleeping_time = 1;
   sleeping_time_increment = 100;
  }

  sleeping_time = min(sleeping_time, frame_interval * 1200);
 }

 capturing_thread_finished = true;
}




int main(int argc, char* argv[])
{
 // spawn new thread that calls capture_frames() function
 thread capturing_thread(capture_frames);     

 //wait in the while loop until the capturing thread set the frame_interval variable
 while (frame_interval < 0)
 {
  //If the capturing thread has finished its execution due to some error, exit the program
  if (capturing_thread_finished)
  {
   capturing_thread.join();
   cin.get();
   return -1;
  }

  this_thread::sleep_for(chrono::seconds(1));
 }

 String window_name = "My First Video Backwards";
 namedWindow(window_name, WINDOW_NORMAL); //create a window

 list<Mat> main_thread_frame_buffer;

 while (true)
 {
  main_thread_buffer_size = static_cast<int>(main_thread_frame_buffer.size());


  //This while loop transfer frames from the shared buffer to the main thread buffer
  while (main_thread_frame_buffer.empty())
  {
   mtx.lock();

   //If there are no more frames to be taken from the shared buffer
   if (capturing_thread_finished && shared_frame_buffer.empty())
   {
    mtx.unlock();
    break;
   }

   //Append frames to the main thread buffer
   main_thread_frame_buffer.splice(main_thread_frame_buffer.end(), shared_frame_buffer);
   
   mtx.unlock();


   //Waiting until the shared buffer is filled by the capturing thread
   if (main_thread_frame_buffer.empty() && waitKey(1) == 27)
   {
    cout << "Esc key is pressed by user. Stoppig the video" << endl;
    break;
   }
  }

  //If the above while loop does not fill the main thread buffer, break the loop.
  if (main_thread_frame_buffer.empty())
   break;

  //Take out the first frame of the main thread buffer
  Mat frame = main_thread_frame_buffer.front();
  main_thread_frame_buffer.pop_front();

  //show the frame in the created window
  imshow(window_name, frame);

  //wait for for frame_interval in milliseconds until any key is pressed.  
  //If the 'Esc' key is pressed, break the while loop.
  //If any other key is pressed, continue the loop 
  //If a key is not pressed within the frame_interval time duration in milliseconds, continue the loop
  if (waitKey(frame_interval) == 27)
  {
   cout << "Esc key is pressed by user. Stoppig the video" << endl;
   break;
  }
 }

 main_thread_finished = true;
 capturing_thread.join();
 
 return 0;

}

Copy and paste above code snippet into your IDE and run it. Please note that you have to replace "D:/My OpenCV Website/A Herd of Deer Running.mp4" in the code with a valid location to a video file in your computer. Then you will be able to see your video file playing backwards.


Explanation


Let's go through the above code line by line.


//Uncomment the following line if you are compiling this code in Visual Studio
//#include "stdafx.h"

#include <opencv2/opencv.hpp>
#include <iostream>
#include <thread>   
#include <mutex> 
#include <atomic>

using namespace cv;
using namespace std;
These are the include files and namespaces used in our program. Please note that std::thread, std::mutex and std::atomic classes are supported after C++11.


//Mutex used to protect the shared_frame_buffer variable
mutex mtx; 

//The capturing thread will insert video frames to this list 
//Main thread will take out video frames from this list
list<Mat> shared_frame_buffer; 

//This boolean will be set to true after the capturing thread finishes its execution
atomic<bool> capturing_thread_finished = false;

//This boolean will be set to true after the main thread finishes its execution
atomic<bool> main_thread_finished = false; 

//The time interval in milliseconds between consecutive frames to achieve the specified frame rate
atomic<int> frame_interval = -1; 

//Size of the list in the main thread which contains frames taken from the shared_frame_buffer
atomic<int> main_thread_buffer_size = 0; 
These are the global variables used in this program. These variables should be declared as global because they are accessed from both threads.

Let's examine the main thread first before going through the capturing thread.


Execution of the Main Thread


// spawn new thread that calls capture_frames() function
thread capturing_thread(capture_frames);     
Above line will spawn the capturing thread and execute the capture_frames() function.


//wait in the while loop until the capturing thread set the frame_interval variable
while (frame_interval < 0)
{
 //If the capturing thread has finished its execution due to some error, exit the program
 if (capturing_thread_finished)
 {
  capturing_thread.join();
  cin.get();
  return -1;
 }
 this_thread::sleep_for(chrono::seconds(1));
}
The above code segment will execute the while loop until the capturing thread set a positive value to the frame_interval atomic variable.
If the capturing thread finishes its execution before setting a positive value to the frame_interval, the main thread should also exit.


String window_name = "My First Video Backwards";
namedWindow(window_name, WINDOW_NORMAL); //create a window
The above 2 lines of code will create a new window identified with the name "My First Video Backwards".


//This while loop transfer frames from the shared buffer to the main thread buffer
while (main_thread_frame_buffer.empty())
{
 mtx.lock();

 //If there are no more frames to be taken from the shared buffer
 if (capturing_thread_finished && shared_frame_buffer.empty())
 {
  mtx.unlock();
  break;
 }

 //Append frames to the main thread buffer
 main_thread_frame_buffer.splice(main_thread_frame_buffer.end(), shared_frame_buffer);
 
 mtx.unlock();

 //Waiting until the shared buffer is filled by the capturing thread
 if (main_thread_frame_buffer.empty() && waitKey(1) == 27)
 {
  cout << "Esc key is pressed by user. Stoppig the video" << endl;
  break;
 }
}
This code segment will take all the video frames out from the shared buffer and insert into the buffer in the main thread. This code segment will execute only when the buffer in the main thread is empty. If the shared buffer is also empty, it will iterate in the while loop until one of the following conditions are met.
  1. Capturing thread fill the shared buffer with some video frames.
  2. Capturing thread is finished its execution and the shared buffer is empty.
  3. The Esc key is pressed and the local buffer in the main thread is empty.


//If the above while thread does not fill the main thread buffer, break the loop. 
if (main_thread_frame_buffer.empty())
 break;
If the while loop breaks because of the 2nd or 3rd  conditions, main_thread_frame_buffer will be empty. In such scenarios, the main while loop should also break from the loop.


//Take out the first frame of the main thread buffer
Mat frame = main_thread_frame_buffer.front();
main_thread_frame_buffer.pop_front();

//show the frame in the created window
imshow(window_name, frame);

//wait for for frame_interval in milliseconds until any key is pressed.  
//If the 'Esc' key is pressed, break the while loop.
//If any other key is pressed, continue the loop 
//If a key is not pressed within the frame_interval time duration in milliseconds, continue the loop
if (waitKey(frame_interval) == 27)
{
 cout << "Esc key is pressed by user. Stoppig the video" << endl;
 break;
}
In this code segment, the first frame in the buffer is taken out and displayed in the window. Then the program will wait for a time duration in milliseconds as specified in the frame_interval and continue the loop. If Esc key is pressed during the waiting time period, while loop will break.


main_thread_finished = true;
capturing_thread.join();
After breaking from the while loop, the main_thread_finished boolean variable should be set to true. The capturing thread will read this variable and exit.


Execution of the Capturing Thread


The capture_frames() function will be executed in the capturing thread. Let's go through the capture_frames() function line by line.


//open the video file for reading
VideoCapture cap("D:/My OpenCV Website/A Herd of Deer Running.mp4");

// if not success, exit program
if (cap.isOpened() == false)
{
 cout << "Cannot open the video file" << endl;
 capturing_thread_finished = true;
 return;
}
This code segment will open the video file. If it is failed to open the video file, capturing thread will exit after setting the capturing_thread_finished boolean variable to true. The main thread will read this variable and exit.


//get the time interval of consecutive frames in milliseconds to achieve the frame rate of the video 
frame_interval = static_cast<int>(1000 / cap.get(CAP_PROP_FPS));
This line will obtain the frame rate of the video and calculate the required time interval in milliseconds between consecutive video frames to achieve this frame rate.


//get the total frame count of the video
 int remaining_frame_count = static_cast<int>(cap.get(CAP_PROP_FRAME_COUNT));
 
 //sleeping time in microseconds
 int sleeping_time = 1;
 int sleeping_time_increment = 100;

 //High buffer size - Memory consumption will increase, CPU consumption will decrease
 //Low buffer size - Memory consumption will decrease, CPU consumption will increase
 int buffer_size = 20;
This line segment declares and initializes variables used in the capturing thread. Usages of each variable are commented inline.


//Finding a starting frame number to start decoding the frames in the video
int starting_position = remaining_frame_count - buffer_size;
  
if (starting_position < 0)
 starting_position = 0;

//Setting the starting frame number to be decoded next in the video. 
//This function call is expensive in terms of CPU consumption.
cap.set(CAP_PROP_POS_FRAMES, starting_position);
This code segment lies in the top of the while loop. It will set the position of the video to be decoded next in each iteration of the while loop.

In each iteration, number of frames which is equal to the buffer size will be decoded. Therefore if the buffer_size is higher, the number of calls to the cap.set(CAP_PROP_POS_FRAMES, starting_position) function  will be decreased and vice versa.


for (int i = starting_position; i < remaining_frame_count; ++i)
{
 Mat frame;
 bool bSuccess = cap.read(frame); // read a new frame from video 

 assert(bSuccess); //bSuccess variable should be always true

 local_frame_buffer.push_front(frame);

 //This line sleeps for a time duration in microseconds as specified in the sleeping_time variable.
 this_thread::sleep_for(chrono::microseconds(sleeping_time));
}

remaining_frame_count = starting_position;
In this for loop, the capturing thread reads frames from the video of which frame number is in between starting_position and remaining_frame_count and adds those frames to a local buffer.

In each iteration, the capturing thread will sleep for some calculated time to control the speed of execution of the thread. If the speed of execution of this thread is higher, the frames will be decoded at a rate more than required for the main thread. If the speed of execution of this thread is lower, main thread may have to wait longer time interrupting the video playback.



mtx.lock();
shared_frame_buffer.splice(shared_frame_buffer.end(), local_frame_buffer);
int total_frame_buffer_size = static_cast<int>(shared_frame_buffer.size()) + main_thread_buffer_size;
mtx.unlock();
This block of code transfers frames in the local buffer to the shared buffer. The shared buffer is protected with a mutex because the main thread also accesses the shared buffer.

The above code segment also calculates the total number of frames in the memory at the moment.


//This section will control the speed of execution of this thread
if (total_frame_buffer_size > buffer_size * 3)
{
 sleeping_time = sleeping_time + sleeping_time_increment;
 sleeping_time_increment = min(5000, sleeping_time_increment * 2);
}
else
{
 sleeping_time = 1;
 sleeping_time_increment = 100;
}

sleeping_time = min(sleeping_time, frame_interval * 1200);
The above code segment adjusts the value of the variable sleeping_time according to the total number of frames in the memory. If the total number of frames in the memory is higher than the 3 times of the buffer size, sleeping_time will be increased gradually up to (frame_interval * 1000 * 1.2) microseconds to slow down the execution of the capturing thread. If the number of frames in the memory is lower than the 3 times of the buffer size, sleeping_time will be set to 1 microsecond to speed up the execution of the capturing thread.

The value of the sleeping_time_increment determines the number of microseconds by which the sleeping_time is incremented at a time.


This is all you need to play a video file backwards. Try changing the values of buffer_size, sleeping_time, etc variables and observe the CPU usage, memory usage and the video playback.

If anything unclear, please leave a comment.


Next Tutorial : Write Image & Video to File


Posted by Shermal Fernando