In the previous tutorial we learnt about how HTTP can be used in IOT ie. for use-cases where a sensor needs to send data to the server at regular intervals. We learnt about a backend framework - Flask and how it helps us serving the API endpoints for GET and POST requests on the server. Then we discussed about a real-world problem, that occurs when we wish to update that data in real-time on a frontend(browser). This covers some depths of the Web Development world, but is necessary for an IOT developer to know, because as I said earlier an IOT developer must be that jack of all trades.
NOTE: The problem we are discussing here is not only an IOT use case problem, it is a very general issue because in any Web system there could be a need to render real-time data on the frontend, it may be a dashboard or any simple website hardly matters the concept is always the same.
Real-time update sensor values on Dashboard - The most efficient way
Flask and mostly all backend frameworks have the ability to create and open socket connections, using a wrapper around the Websockets protocol Flask comes with a library called Flask-SSE SSE stands for Server Sent Events and is the best possible solution to the problem of updating real-time values on the Frontend. We will see it in practical, in a bit.
In this post we are going to explore quite deeply how Flask_SSE works for realtime data streaming and how Javascript helps us here to grab that data. We will also discuss use-case scenario in which we can use AMQPS broker and Database such as redis for more efficiency. So let's get started.
Server Sent Events with Flask Example
Architecture
What we will do now is create a very basic Flask application to render current date and time on the frontend in real-time. The Flask backend will stream the current date and time every second and a small Javascript will use that data and render it on the DOM.
Now let's see this in action.
Steps for Setting up the Application
i. Activate the virtualenv created in Part-1.
ii. Clone the repository code using git, if git is not install please install it, it's a simple installation for both Windows/Linux or MacOs, just Google search on how to install git on your OS. Assuming that git is installed run,
git clone https://github.com/RtjShreyD/SSE-with-Flask.git .
iii. After successful run of above command a new directory would have appeared named SSE-with-Flask.
iv. Move into the cloned project by running, cd SSE-with-Flask and then install the requirements, run pip install -r requirements.txt.
v. Once the requirements are installed, run python app.py to start the server.
vi. Now open browser and goto localhost:5000, what we see is that the current date and time is being updated in real-time on the frontend.
The application here is using the concept of SSE in Flask and giving you real-time data on frontend. Basically, through SSE any data generated on the backend can be configured as SSE, which can be picked by Javascript's EventSource() function. Let's take a look at the code.
We have a dir structure like below
SSE-with-Flask
|
|-- templates/
| |
| |-- index.html
|
|-- app.py
|-- .gitignore
|-- README.md
|-- requirements.txt
Now all the game is being played by app.py file and the index.html file. We should discuss about it, so examining the code of app.py, we have defined two routes - / and /stream, the /stream route runs a function named eventStream() which runs an infinite loop yielding the current date and time every 1 second. The data yielded by eventStream() function is returned as a response on the streamed socket connection.
Next we have the index.html file which runs a Javascript on a <div> and connects to the /stream route on server through sockets internally, the connection and receiving streamed data from the connection is handled by the EventSource function in Javascript, which just manipulates the InnerHtml of the div element of the DOM as specified.
This way data generated at the server side is send and updated on the client-side(frontend) in real-time. So this was a very basic example to demonstrate the concept of SSE, however we will be looking at a more typical example of SSE with an actual use-case next.
In the next practical we will be looking at the same use-case as done in Part 1, where a server hosts an API endpoint for sensor data, an IOT device or any client can send data to server as POST requests and then the data sent is displayed in real-time on the frontend. This is a common use-case of any Dashboard not only just IOT. So let's see it in action.
Architecture for Flask-SSE with redis for real world use-case
Steps for Setting up Flask-SSE with redis for real world use-case
If you have Docker and docker-compose installed you can follow the Run with Docker steps in the README.md after cloning this repository, else follow the steps below and don't get confused about Docker and related stuff for now.
i. Install Redis, following the redis docs.
ii. Clone the repository run,
git clone https://github.com/RtjShreyD/iot-flask-redis-sse.git.
iii. Create and Activate python virtual environment.
iv. Install requirements run,
pip install requirements.txt.
v. Run Redis command in new terminal, redis-server.
vi. Run App, python app.py and watch it on localhost:5000.
vii. Using Postman or any tool send POST request with header content-type --> text-plain and send any numerical string to route /therm/push.
We can send a POST request from a new terminal as well by running,
curl -X POST -H "Content-Type: text/plain" -d '{"temperature":56}' http://localhost:5000/therm/push.
viii. The value should get updated immediately on frontend without reloading it. Repeat the above step with different data and you can see that update on frontend in real-time.
ix. To stop the redis-server, press Ctrl+C or if the terminal got closed by mistake Redis would still be running, so run redis-cli shutdown to close it.
Now that we have successfully established real-time communication between the server and client(browser) with and without redis, we should be wondering what is redis and what is it doing over here. So let's discuss it, Redis is actually a Database which supports key-value pair data storage as well as it is a messaging broker. Here using redis gives us two benefits - firstly that now our data shall be stored in a key-value data store, in a local file with .rdb extension and secondly it allows other systems/clients/servers which interact with our main server, say an analytics server also needs that streamed data to perform analytics on the data-history or any similar application. If we look at the architecture diagram carefully we see
External Client or Server , this is the guy who is a 3rd party and needs to use the streamed data for any purpose, so redis serves this as well. However in heavy real-world systems, using separate Database instance and a separate messaging broker is preferred for various reasons. We will look deep into messaging brokers when we move forward to MQTT and AMQPS use-cases in IOT soon, then we may have a clear understanding of what to use when.
NOTE: In this particular use-case which we have discussed with the practical here, we have just configured a redis database and for now we are not concerned about it's broker aspect at all, however if a 3rd party wants they can connect to the redis broker to use the data.
Congratulations!! With this done, we have successfully learnt about how to create API endpoint for sensor data, how we can POST and GET data from the endpoint and how that data can be displayed and updated in real-time on the frontend. So we are done with the Application side part and now we have to deploy the application on cloud so that any our iot sensor or a client POST data to the server from anywhere. Once that is done we will look into how devices like Nodemcu or ESP32 can send real-sensor data to our server. So see you all in next post.