IOT use-case with Flask and HTTP (Part 1)

Hello developers and builders, we are back here with another simple IOT tutorial, where we will learn how to create a micro-server with Flask framework and upload sensors data to the server and see that update on a browser web page. It's the simple use case of a dashboard, where there can be many sensors and we need real time updates from each sensor at every fixed number of seconds or minutes or so. Here, we are going to simulate this feature with the least possible resources and time to deliver the concept. So let's understand how we will do it.

Firstly we will be creating a basic flask server, to get you breifed with how Flask is used, then we will create a small backend API in Flask to support the POST HTTP request method on an endpoint. The endpoint can be used by our sensor in reality to POST data, however here we shall neither use a sensor nor shall we use a Nodemcu or ESP32 or any other IOT module to interact with our server, infact we will just test the API endpoint via our PC's terminal and simulate the behaviour of a POST request to our server. And we may also discuss some best practices in Flask.

What is Flask ?

" Flask is a micro web framework written in Python. It is classified as a microframework because it does not require particular tools or libraries. It has no database abstraction layer, form validation, or any other components where pre-existing third-party libraries provide common functions. " - Source Wiki).

We can easily develop applications on Flask and it is a kind of DIY framework for web development. For those who want to get introduced to what is backend developement should begin with Flask. The Flask Documentation contains all the necessary information to start learning Flask. You can follow the Installation steps and then jump to the [Tutorials] section in the above documentation, to learn Flask deeply and you can even get started here with Flask by following this tutorial till the end. So let's create a Server with Flask.

Note - There will be separate instructions marked for Windows users.

Creating a Flask Server

Step 1 - Setting up Python environment

For Linux(Debian) users -

Make sure you have pip installed in your system, to check run the following command in terminal,

pip3 --version

It should respond with the current version of pip you are using, this means pip is installed, otherwise not. If pip was not installed, install it by running,

sudo apt-get install python3-pip.
Make a new dir named as flask-iot run,

mkdir flask-iot

and then move into the dir run,

cd flask-iot

We will do all this tutorial in this very dir.
Install a virtual environment wrapper run,

pip3 install virtualenv.
Now that virtualenv was succesfully installed we need to create a virtual environment using the virtualenv wrapper we just installed, run

python3 -m venv flask-env

Note that flask-env is the name of our environment and we are inside flask-iot dir.
Now we will check whether our virtual environment was successfully created or not, for that we first make sure we are in flask-iot dir in the terminal and then we run,

ls

The above command lists down all the contents of the current dir which is IOT-AWS and we should see a dir named iot-env created inside the current dir.
Now we will activate the environment by running,

source flask-env/bin/activate

If the environment was successfully activated on your terminal, you would see (iot-env) before your username in the terminal. This virtual environment is activated and what it means is, that now whatever you will install in this environment via pip will only affect this environment's python interpreter and your global interpreter of Python shall remain unaffected, this ways we can work on diverse type of projects while not messing up with the original Python interpreter.
Now we need to install few things so run the following commands,

pip install -U pip -- this will upgrade the pip version in the virtualenv to latest.

pip install Flask -- this will install Flask and all dependencies.

For Windows users -

Download and install Chocolatey which is a package manager for Windows if it is already installed ignore this step.
Now we need to install Python3 - if already installed ignore this step, using Windows command line or Powershell run,

choco install python --version=3.8.0.
Install pip and upgrade pip run,

choco install pip choco upgrade pip
Install virtualenv wrapper run,

pip install virtualenv.
Make a new dir named as flask-iot run,

mkdir flask-iot

and then move into the dir run,

cd flask-iot

We will do all this tutorial in this very dir.
Create a virtualenv in the flask-iot dir run,

virtualenv flask-env

Note that a dir named flask-env appeared inside flask-iot, now we need to activate the flask-env.
Activate the env run,

\flask-env\Scripts\activate

If the environment was successfully activated on your terminal, you would see (iot-env) before your username in the terminal. This virtual environment is activated and what it means is, that now whatever you will install in this environment via pip will only affect this environment's python interpreter and your global interpreter of Python shall remain unaffected, this ways we can work on diverse type of projects while not messing up with the original Python interpreter.
Now we need to install few things so run the following commands in sequence,
1. pip install -U pip --> this will upgrade the pip version in the virtualenv to latest.
2. pip install Flask -- this will install Flask and all dependencies.

We have successfully installed Flask and now we are done with setting up our system for both Linux and Windows based users. To deactivate the virtual environment we can press Ctrl+D in the terminal(may only work in Linux) or simply run deactivate command in terminal (works on both windows and Linux), only if we need to deactivate it at some point for now we are good to go.

Step 2 - Getting started with Flask

i. Make sure virtualenv is activated, move into flask-iot dir and create a new file named app.py at this location. Windows users need to create the file manually by right clicking > New file > Save it as app.py. Linux users can follow the commands,

cd flask-iot

touch app.py

ii. Copy the following code in app.py and Save it.

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == "__main__":
    app.run()

iii. Now make sure you are in the same dir as the app.py file in the terminal or windows commandline or powershell and run the server,

python app.py

iv. If there are no errors and the server is running, you should see a message like in the following image,

v. If you got the response as shown in above pic, open the browser and open the url http://127.0.0.1:5000 or localhost:5000, both are same actually, you will find the message Hello World is displayed on the browser.

Congratulations !! we have successfully created a flask server and we have a hint of how it works, however now I won't be going to explain all the basic concepts and details here as this course is not about Flask, it is about IOT. However we will cover some basic concepts of how Flask and HTTP REST APIs can be used in IOT systems and before that we will discuss in practical a common mistake which some beginner IOT developers do, with handling IOT with HTTP and also we will discuss how HTTP along with another masterpiece protocol is used for real world applications of IOT within a Flask microservice.

So now we will create a REST API in our server, the purpose of this API is that our sensor which is say a temperature sensor will push data to our server, at fixed intervals and other clients may need to access that data from server as well. This is a quite common Industrial use case, so let's see it in action and deduce our learnings.

Step 3: Creating POST and GET API endpoints for sensor data

Assuming we have a temperature sensor which is accessible in the local network, we will create a POST API endpoint for its data.

i. Let's stop the sensor by pressing Ctrl+C in the command line and copy and paste the following code into our app.py file.

from flask import Flask, request, jsonify
import flask
import json

app = Flask(__name__)

sensor_val = ''

@app.route('/')
def hello_world():
    return 'Hello, World!'

@app.route('/therm/push', methods=['POST'])
def post():

    global sensor_val

    message = request.data

    sensor_val = message

    return flask.Response(status=204)


@app.route('/therm/get', methods=['GET'])
def get():

    global sensor_val

    json_str = sensor_val.decode('utf-8')


    return jsonify(json_str)



if __name__ == "__main__":
    app.run()

ii. We will save the app.py file and run the server again using python app.py.

iii. Now that our API is created, we need to test it out, after testing I'll explain what we have done. Assuming the server is running we will open a new terminal session or command line session and let the previous one running. Next run the following command in the terminal or command line as per OS.

Linux Users Run - curl -H "Content-Type: application/json" -X POST -d '{"temperature":45}' http://localhost:5000/therm/push.

Windows Users Run - curl -H "Content-Type: application/json" -X POST -d {\"temperature\":\45\} http://localhost:5000/therm/push.

iv. Upon running the above command, the terminal running the flask server, should log a POST request in the terminal space, the new entry logged on terminal should look somewhat like the following, 127.0.0.1 - - [02/Apr/2021 00:21:05] "POST /therm/push HTTP/1.1" 204 -.

v. If the POST request was logged as above, this means that the sensor data was successfully received by the server, we could store it in a Database as well however we haven't used any Database yet, so this value of temperature that we are sending to the server will remain until a new POST request is sent with a new value.

vi Next we can try to retrieve the data from the server using a GET request, which we will send via the terminal using the curl utility as we did for POST. In the other terminal or command line run the following command,

curl http://localhost:5000/therm/get.

vii. We should see a response in the terminal like the following, "{\"temperature\":45}" and furthermore, we should also see an entry logged in the other terminal like the following, 127.0.0.1 - - [02/Apr/2021 00:30:17] "GET /therm/get HTTP/1.1" 200 -.

viii. If the above step behaved as expected, we have successfully created and tested two REST API end points in a Flask server, for the sensor data. We can modify the temperature value POSTED in Step 3 and see the updated value in Step 6 response.

Now you might be wondering, what has just happened here. So we will try to understand this better. What we did just now is a simple IOT use case where a client device generates certian data, here in this case we assumed it to be a temperature sensor, and the sensor device upon connection with LAN or Internet sends the data to the server at fixed intervals. There can be other client devices or cients browsers which need to view the sensor data as soon as it is received. In a practical use case there can be a NODEMCU interfaced to a temperature sensor sending data at regular intervals using POST API requests and that data can be seen on a browser web page or a dashboard application. We are just simulating the use case here. So according to HTTP we can send data through a POST request which we have successfully done and to retrieve the data we use a GET request which also we have tested. However as this is just a simulated use-case and in actual product there could be other things involved as well such as database, authentication and frontend design, cloud server, etc. We are not going into that depth for now as we just are concerned about how IOT works with HTTP and Flask is just serving as an example here.

So we have successfully tested that we can POST sensor data to a server and retrieve that data by sending a GET request to the server. Now suppose we want to build a system where this sensor data should be updated on frontend(browser) in real-time. Sounds easy, right, this is the most common use-case where different people give different solutions to the problem. Let's discuss a few solutions and then we can see the best one working out for us in practical.

Real-time update sensor values on Frontend - Solution 1 (Polling)

A scenario can be where a device POSTs data to server at regular intervals and then the frontend(HTML+CSS+JS) on browser retrieves and display that data on web page by sending GET requests to the server at regular intervals and as soon as a value different from the previous value is received it just displays it. Seems easy to implement and is really very easy, however, this method has some drawbacks, let's discuss those. Such a system is called a Polling based system, in which a client(browser in our case) sends repeated requests to the server, to check if updated data us available or not, this leads to a lot of unnecessary load on the server and in production it also hugely affects the costing to using cloud server instances as all the service providers like AWS. DigitalOcean, etc have limited number of free requests on server, which if exceeds there are additional charges. Now in a polling based system a lot of requests can just go waste, in checking whether new data has arrived or not.

Let's take a more day to day example to understand this problem, suppose you are attempting to solve a difficult math problem and your teacher stands on your head and asks you every 10seconds "Did you solve it ?" and you replying to him "No" every time untill you have solved it really. Can you imagine how annoying it would feel if somebody bugs you every now and then asking whether its done or not. Its now a real life problem how to solve it, a best optimal solution would be that the teacher gives you the problem and asks you tell him only when you have solved it. This ways for the time you haven't solved it the teacher would know it is unsolved and the moment you solve it you tell him yourself that it is solved, without him asking again. Seems a legit algorithm, however for this algorithm to work, the student should be in contact with the teacher the whole time he was solving the problem, then only the moment it got solved the teacher will immediately know.

Now let's think how to do it in our case with the sensor, server and browser. As we know an HTTP request is open on the server only for the time it is getting serviced, once the request is services ie the response was received to the client, the request gets closed and a new request may get open. So no matter how fast the server does the processing and parse that data, server can't send that data to the browser untill there is a request for it, so this can't be possible in real-time with HTTP protocol becasue HTTP is just a short lived connection to the server. In order to do it efficiently we need a long lived connection. Now to establish a ling lived connection we need something else like - WEBSOCKETS.

Real time update sensor values on Frontend - Solution 2 (Sockets)

The above problem is solved fully if the browser remains always connected to a process on server which immediately gives it the data as soon as updated data is received on server via a POST request. Here Websocket protocol helps us in establishing a long-lived connection to the server, which helps in making the system more efficient. The idea is to have the POST API route on server as it is and instead of GET API endpoint the server should open a Websocket port with which the frontend Javascript can connect and be connected, thus it can in real-time receive and display the updated data on the web page.

In the next blogpost we will discuss in depths about the best solution for the problem and see thing happening in practical.

In the next post we will practically see how it works efficiently in the best possible manner.