System design is a critical aspect of software engineering that involves identifying user requirements, analyzing problems, defining system architecture, and developing a prototype. The goal should be always to create a robust and scalable system that meets the needs of the users. In this note, we will discuss the key components and processes involved in architecting software.
Keep in mind that every project and company have their own limitations. And we, as humans, also have our limits defined by how much we know about the business and the technologies involved. Problems may create limitations, and some of these limitations may get really hard to overcome.
To illustrate the concepts discussed, we will use a case study of a system that extracts thumbnails from live streams and inserts them into a database to be queried further. The system design will address scalability and reliability concerns, and the database used to store thumbnail data will be agnostic.
Youtube, for example, shows in the seek bar a thumbnail with the exact image the program you were watching, you just need to hover your mouse over the seekbar. It is a very useful feature for live stream you can watch the past minutes.
Identifying user requirements
The first step in system design is identifying user requirements. In the case of our thumbnail extraction system, the user requirement is to extract thumbnails from live streams and store them in a database. Users should be able to query the database for specific thumbnails based on parameters such as the time the thumbnail was captured, the stream it was captured from, and other metadata.
Analyzing the problem
Once we have identified the user requirements, the next step is to analyze the problem. We need to consider factors such as the number of streams the system will handle, the frequency of thumbnail extraction, and the amount of data that will be stored in the database. For example, the number of the stream and the thumbnail quality will impact the amount of storage we will need, increasing the cost and the latency of our queries.
Defining system architecture
After analyzing the problem, the next step is to define the system architecture. In the case of our thumbnail extraction system, the architecture would consist of a set of components that work together to make available all the stored thumbnails to users.
- A component that process live streams and extracts thumbnails from them.
- A component that inserts all extract thumbnails in a database.
- A query component that enables users to query the database for specific thumbnails based on parameters like datetime, width and height.
Diagrams are tools to facilitate the understanding of how things get connected.
In my experience, beginners have less difficulty trying to understand complex systems with visual information.
Developing a prototype
Once the system architecture has been defined, the next step is to develop a prototype. The prototype should be designed to test the system architecture and identify any potential issues or limitations. In the case of our thumbnail extraction system, the prototype would involve developing and testing each component of the system.
Designing a system involves a critical stage where we get closer to reality, making it one of the most crucial parts of the process. Although in theory all our ideas should work, it is common to face challenges and iterate in order to find more accurate solutions to problems.
A concrete example: to extract thumbnails, one of the best tools to do the job is ffmpeg. And how does ffmpeg extract thumbnails? Ignoring the multitude of options and flags available, ffmpeg saves every thumbnail in the filesystem. Would it be easier if ffmpeg could send all the files through an API? Problably, but we would end with different problems: network connections are more susceptible to fail, and they will be much slower than writing a file to the disk. Since we are doing with realtime systems, we need to account every time we lose passing information around.
Iterating
Asking questions is one of the most valuable ways to find answers. Ask, for example, how long our thumbnails will stay in the database? Since these are thumbnails from live streams, do they need to live forever? If not, how could we remove old thumbnails? Using strategies like defining a TTL to every thumbnail might be an alternative to solve the problem. Then it brings other questions to the table: how to query the database? Which structure are we going to use? Defining the right data structure to the problem may be a hard job but it will pay the time we take to choose them more precisely.
Other topics to consider
Keep in mind that the ideas mentioned above are general considerations to apply when designing a system. However, there are many other essential topics you should be concerned about, such as scalability, monitoring, performance, cost optimization, and more. Designing a robust system is a highly complex task that requires careful attention to various aspects.
I will talk more about these topics over the time.