Introduction
Welcome builders and developers to this complete IOT tutorial series. As we had discussed in the [previous blogpost] about what is Iot and we know that in Iot hardware and software is linked together via Internet. In this tutorial and some upcoming ones we will learn about the Web and Iot protocols that are used to connect devices together. This one shall be a complete theory tutorial and is not only for Iot enthusiasts but also for web development eenthusiasts, because as I mentioned in earlier post that there is a very thin line which separate Iot development to Web development technologies and we will see how. In next tutorials after this one we will look practically how these protocols work using Python because all single board computers used in IOT like Raspberry Pi, Orange Pi, Beaglebone, etc support linux based OS and are easily programmable with Python. We will take up tutorials also with C language using Nodemmcu and Esp32 embedded devices in later posts, so bear with me guys and girls and let's get started.
The HTTP protocol
HTTP and HTTPs basically is thr most widely used web protocol without which Internet was not possible, the whole Internet is running on it and every website or application that we use, uses HTTP and HTTPs so what is it ?.
According to Mozilla Docs “HTTP is a protocol which allows the fetching of resources, such as HTML documents. It is the foundation of any data exchange on the Web and it is a client-server protocol, which means requests are initiated by the recipient, usually the Web browser. A complete document is reconstructed from the different sub-documents fetched, for instance text, layout description, images, videos, scripts, and more. Clients and servers communicate by exchanging individual messages (as opposed to a stream of data). The messages sent by the client, usually a Web browser, are called requests and the messages sent by the server as an answer are called responses.” HTTP Protocol comes with a facility called “requests/responses”, each communication that takes place on Internet with HTTP, involves the play of requests and responses. Requests are like a function which runs over the internet, takes in an endpoint, may or maybe some arguments and run, returning a response. Endpoint is the combination of IP and Port n.o which is mapped to a custom domain name, it looks like “http://180.90.20.206:8000", where former is the IP and later is the port no, it can be mapped to a domain like eg. example.com Requests are the medium to send data. The API accepts or sends requests, which are of GET,POST,PUT,DELETE, as name suggests GET is mainly for fetching useful data from api endpoint, POST is mainly for sending data to api endpoint and PUT for replace and DELETE for deletion of data from database at the server which serves the endpoint. The image below would explain well about HTTP protocol, with requests and responses.
So now we have a basic understanding of how, HTTP works, but a small problem with HTTP is that connection is not secured, to make it secured we have an SSL (Secured Socket Layer) certificate, which encrypts the traffic end to end transfer of the data. Such web services have a url starting with https instead of http. So you must be wondering how the data in received or sent in requests, basically we are going to discuss about the format of data and how that data is acknowledged at an endpoint. So there are 2 things basically,
API
API is the acronym for Application Programming Interface, which is a software intermediary that allows two applications to talk to each other. We should also learn about what is Json - JSON (JavaScript Object Notation) - JSON is a lightweight format for storing and transporting data. It has curly braces and include data in key value pairs.
REST API
Wiki definition is “Representational state transfer is a de-facto standard for a software architecture for interactive applications that typically use multiple Web services. In order to be used in a REST-based application, a Web Service needs to meet certain constraints; such a Web Service is called RESTful.” Statelessness in REST APIs. Statelessness means that every HTTP request happens in complete isolation. When the client makes an HTTP request, it includes all information necessary for the server to fulfill that request. The server never relies on information from previous requests. If that information was important, the client would have sent it again in this request. Use Json format. The below image shows a REST API in json format supposedly for a user, it also denotes the key value pairs used in json.
We will have an in depth understanding of how HTTP is used IOT systems, when we will do a small project, using Flask and ESP32, where we will upload simulated sensor data to an IOT server built on Flask
Websockets protocol
As we learnt about HTTP, we learnt that it was completely based on request response architecture, and the connection was shortlived meaning to say that the connection exists only till the request was being serviced, and then the connection is lost. Now in web systems there are many use cases where a continous long lived connection is required, so there are certian protocols which provide long lived connections and depending upon the use cases we choose one. Websockets is one such protocol which provides a long lived connection in p2p architecture, p2p means point to point architecture, ie a client can get connected to the server till the time it wants to remain connected and bi-directional exchange of data can take place in that time. Websockets and HTTP both work over the TCP/IP communication layer. We will have a practical implementation of websockets in further tutorials with Python, then we shall deeply understand how they work, however websockets are not very widely used in IOT systems but highly useful in Web systems.
The MQTT protocol
This is the most widely used protocol in IOT and is the backbone of IOT and connected devices. MQTT stands for Message Query Tansfer Telemetry, it is a very light-weight protocol which provides a long lived connection. MQTT is different from Websockets in a lot of ways, however the major difference is that in MQTT the connection is not a p2p connection, which means at the same time a lot of devices can be connected through the same TCP/IP communication channel. We will understand MQTT with the following figure,
In the above Fig we can see MQTT protocol and HTTP working together in a probable real world system. If we see the red and orange colour index it points to Publish and Subscribe. MQTT is based upon a Pub/Sub model. As we see in picture there is a cloud Broker, this broker is a server basically which routes the messages from one node to another based on their preferences. Each node in system, can directly send and receive messages to and from the broker. Every special class of data is grouped into topics, on the broker, a client node whether be a PC or a microcontroller can Publish data onto a particular topic and can also subscribe the data from various available topics.
Whenever a device needs to send some data to another device or a server for processing, it will Publish that data onto a topic which is defined on the Cloud MQTT broker, another device or the processing server or the mobile application which wishes to receive a certian type of data whenever it is generated will, Subscribe to the topic provided by the Broker, this ways whenever a data is published it is immediately received by all the subscribers of that data. The Broker simply routes all the published messages to the subscribers of those type of messages. In the figure above the MQTT broker routes the sensor data to mobile app, web app and processing server which can further process it and store it in Database. The processed data from the Database can be accessed by other clients simply through HTTP protocol this ways both MQTT and HTTP work together in a real world system.
An important point in MQTT is something called QOS (Quality of Service).This QOS is a parameter which defines how the message will be received by the subscribers, as MQTT faces a small drawback due to network and communication channels, which is sometimes messages may get lost and may never be received or the same mesage may be received more than once. To ensure that the messages are consistently delivered, MQTT broker offer a Quality of Service agreement. QOS has three levels - 0,1 and 2 which state the delivery of messages as Atmost once , Atleast once and Exactly once respectively. Some brokers offer all three levels of QOS while some just offer two ( 0 and 1). Please read more about QOS from here.
There are many Open Source MQTT Brokers available to use, some of the most common are - Mosquitto, HiveMQ, AWS Iot Core, etc.
We will look into MQTT quite deeply in practical and work with Mosquitto and AWS Iot Core, in further tutorials because this is the base of all IOT and we need to play around MQTT all the time while pursuing this course for IOT or whenever we are dealing with IOT and connected devices.
The AMQP
The AMQP stands for Advanced Message Queing Protocol and is a bit similar to MQTT and used in both IOT and Websystems or say more precisely in used cases where the IOT system is just a part of large scale Web system. AMQP is also based on Publisher/Subscriber model as in case of MQTT, this is what makes them similar, however the major difference between MQTT and AMQP is that in AMQP data is never lost. Even if the client is not present to grab the data or say the client is not subscribed still the data remains at the Broker in a queue and whenever the client subscribes it will receive all the data in the queue one by one. In AMQP the acknowledgement of data received is mandatory by the client, thus ensuring reliability and consistency. AMQP is suitable for high amount of data transfer unlike MQTT which is genrally used for low level of data transfers. The Queue is most important thing here which makes it so advanced level protocol.
AMQP is created as an open standard protocol that allows messaging interoperability between systems, regardless of message broker vendor or platform used; With AMQP, you can use whatever AMQP-compliant client library you want, and any AMQP-compliant broker you want. Message clients using AMQP are completely agnostic.
The most common AMQP brokers available are - RabbitMQ, Apache Kafka, ZeroMQ, Amazon SNS, etc. We will have a practical demonstration of AMQP and Cloud AMQP using RabbitMQ in further tutorials.
WebRTC
WebRtc, stands for Web Real Time Communication, it is not a protocol but an open source project and a specification which allows real-time communication over simple web application programming interfaces (API). This follows a p2p architecture and is widely used in all types of video chat and video conferencing based systems. It is best suitable for live video and audio transmission. It has use cases in IOT as well for streaming live feeds from drone cameras and robots.
We won't go in much theoretical details of it for now, however we have planned a nice tutorial on WebRTC in the future posts where we will see it working in a live example use case and also create a small project around it.
Conclusion
Now that we have gained some theoretical background on technologies and protocols that we are going to use in our journey with IOT and Web systems, so I shall end this post here and let's get ready to some practical stuff with these protocols in the next tutorials.
We will be using
Thankyou and Happy learning !!!