IoT-AR | Augmented Reality and Internet of Things; Smart Home

This is my Senior Design Project report which is chosen Best Senior Design Project in Spring 2018

For All Files and Docs, you can visit:

Table of Contents


    2. HARDWARE 5
      1. Equipment 5
      2. Applications 7
    3. SOFTWARE 10
      1. Augmented Reality 10
      2. Stations 11
      3. Website 11
      4. Server 12
      5. Database 13


Augmented reality (AR) and the internet of things (IoT) are trending technologies and will revolutionize the next generation of the industrial products. Various researchers and industrial companies develop applications and devices for the smart home.

In this project, the power of augmented reality combines with IoT devices to improve the user experience in smart home context. A user becomes capable of controlling different sensors installed in a model home with augmented reality. Each room of the prototype has a station with sensors. These stations are connected to a gateway using a unique network. The platform for managing the costume network is Raspberry Pi 3. Data from different stations collected in a database and the user able to monitor these data on a dashboard website. The AR smartphone application will detect known features and the hidden and virtual controllers of sensors or stations will pop-up on the user’s screen. The output of the project improves the user experience and comfort.


In this project I have started to divide my system into 5 main step which is augmented reality application, designing dashboard website, creating a new unique network, designing cloud service architecture, optimizing the database. For the augmented reality application, I decided to use Vuforia library for the texture recognition. Creating website back-end I decided to use Node.js, in the

front-end multiple library and language used which are, Bootstrap, CSS, HTML, javascript. MQTT library provides the cloud service and connection for the internet of things concept.

Manipulating system files on Raspberry Pi 3 provide a new wireless network connection. Storing data in the database provided by MongoDB No-SQL database for Big Data concept.

System Design

2.1 Methodology

Firstly, I had to decide which Wi-Fi module and microprocessor to use for designing stations.

It is important for the project’s processing. After some researching I decided to use Nodemcu ESP8266 microprocessor board because this board has internal Esp8266 Wi-Fi module can programme with using Arduino IDE but I must flash and reboot board with different firmware. After that, I can use Arduino IDE for coding when downloading board library.

Another important thing the communication each board to each other and server. After some researching, I decided machine to machine (M2) connectivity protocol. MQTT is the one of the M2M protocol and it is compatible different platform and programming language, for this reason, I will use MQTT. I draw network topology for use in this project at Figure 2.1.1. MQTT protocol can provide server/client, client/client and peer to peer connection. I will use the server/client connection. In my design server listen to all client. In this system, each client subscribes to a specific topic and can publish data on any specific topics. When the server starts to running listen to specific Socket(IP: PORT) after that if any connection established on this socket with using MQTT, the client send to subscribing topics so that server knows every client specific subscribing topics. When any client publishing message with topics, server send this message to same topics subscriber. This connection’s example showing at Figure2.1.2. In this figure, Sensor reads temperature 27 C and publish this value in the message with “smart home/kitchen/sensors/temp” topics. MQTT server sends this message if any client subscribes “smart home/kitchen/sensors/temp”, “smart home” “smart home/kitchen” “smart home/kitchen/sensors” topics. Website listening this topic and broker send the messages to the website server and website server monitor this value on the web page.

The software part of this project will be based on the MQTT communication protocol but in this project, I use different programming language for the different purpose. In Figure 2.1.3 I describe programming languages working for each platform. Augmented Reality part will be developed with Unity and Vuforia. Firstly, I created image targets which are a specific image for each station or device. After that, I draw control panel designs on specific image targets.


Figure 2.1.1


Figure 2.1.2

Figure 2.1.3

2.2 Hardware

This project has 4 different components which are, kitchen and outside sensor station, smart plug and RFID card reader for door security. I tried to use cheap and useful components. In the sensor station and RFID card reader used with Nodemcu ESP8266 board for the microprocessor. This microprocessor is more useful and cheaper than Arduino Uno for this project concept. For the Smart plug, I decided to use cheap and modified relay board with ESP8266 wifi module.


Nodemcu Esp8266

Nodemcu has its own WiFi module and also have 1 Analog I/O and 8 Digital I/O. Also,

Nodemcu provides a deep sleep system by using a timer. The NodeMCU is a development card having the ESP8266 WiFi module with the firmware installed. Developed using the ESP8266 SDK, the Extra supports GPIO, PWM, IIC, 1-Wire and ADC connections without the need for a microcontroller. On the CP2102 USB – Serial converter is integrated. I used this module for communication with each other and with the server.


Rain Sensor

When this sensor panel is wet resistivity is increased and therefore current is decreased. I read that current value and in the Arduino IDE, I mapped these values with our determined intervals. I read analogue value by using Nodemcu analogue input part.


When light shows up on the LDR resistivity decreases so current is increasing and I read the

current with Nodemcu analogue input port and after limiting this value (output) We use the value for Light’s value.

MQ-9 Gas Sensor

This kind of gas sensor needs more current and voltage and needs warming up for detecting

data so that we connected the extra power supply with USB cable like DHT11. This sensor’s output gives 10bit value 0-1024. If gas detected output give higher voltage.

DHT11 (temperature and humidity sensor)

We used this sensor with its own Arduino library code and these sensors have Analog output.

We connect to extra voltage to this sensor for clean data.


This sensor works like rain sensor. When this sensor panel is wet resistivity is increased and therefore current decreases. We read these current values and in the Arduino IDE, We mapped them with our determined intervals. We read analogue value by using Nodemcu analogue input part.

Flame Sensor

This sensor gives only digital output 0 and 1 if IR sensor detects flame give 1 else 0. Also, its threshold can be changed with its own potentiometer.

Water Level Sensor

This sensor gives only analogue output. This works like rain and moisture sensors i.e. by changing resistance.

RFID Card Reader

A radio frequency identification reader (RFID reader) is a device used to gather information from an RFID tag, which is used to track individual objects. Radio waves are used to transfer data from the tag to a reader. In this reader, I read the RFID cards Hash Code for the determine the card is authorised or not.


The ESP8266 WiFi Module is a self-contained SOC with integrated TCP/IP protocol stack that can give any microcontroller access to your WiFi network. The ESP8266 is capable of either hosting an application or offloading all Wi-Fi networking functions from another application processor. Each ESP8266 module comes pre-programmed with an AT command set firmware.

However, I re-programmed this module for my cloud system

Relay Board

5 Volt 1 Channel Wifi Relay Card. It is a module with one relay on it. There is a 5-volt signal input pin for controlling the relay on the card. The key status can be adjusted by relaying the signal from here. Thus, both AC and DC currents can pass through it. But I modified this board I change pins the trigger the relay.


Network Access Point

As for “Custom Network Access Point”, I will have to encode the embedded WiFi adapter which is in Raspberry Pi 3 as a transmitter, not as a receiver. And it has to make multiple connections work the same way on the same network. There are explanations about this on the Internet but there are other problems in most of the solutions I encountered and I thought that using raspberry as both Web-Site and MQTT server and also as Internet provider reduced the cost and make the system computerless.

This access point provided the wireless internet for ESP8266. It provided more security for the stations also this system is shared on the external connection when LAN connection provided on Raspberry Pi 3. After that, all system and website shared on the external network.


Kitchen Sensor Station

Kitchen sensor station has 4 sensors these are Flammable gas, fire, water level, temperature and humidity. I use the diodes for the controlling analogue inputs. Because nodemcu have only one analogue pin.

Outside Sensor Station

Outside sensor station has 4 sensors these are Rain, moisture, light, temperature and humidity. I use the diodes for the controlling analogue inputs. Because nodemcu have only one analogue pin.

RFID Card (Door)

I used the MFRC522 RFID card reader board. In this board use serial communication with the I2C protocol with the microprocessor. I used the interrupt, I mean board waited to RFID card reading.

Smart Plug

I used the wifi relay board with ESP8266 but I modified the board because this board is not working with my cloud system. This board use serial communication with the relay. I change the triggered pin the relay. I unsolder the 15F104 serial communication chip.

2.3 Software

Augmented Reality

I used the Unity for creating Augmented Reality application. I design the control panel for monitoring the sensors values. Also, I used the Vuforia for the texture recognition. Most challenging

part of this project is MQTT connection because MQTT library for the Vuforia does not exist so that, I create new library and assets for the Vuforia. This Augmented Reality is Marker-based. I design a marker for all stations. Also, I design the model home this home gives feedback for changing sensor and RFID status. For example, when is door unlocked, AR home shows kitchen. Bellow figures show the different control panels and model home. In the kitchen and outside control panels when the sensor values change sensor tags colour

changes and give the feedback also when the user pushes the bellow button,

the user can control the delay and sleep time in the microprocessor. Also, users can use the door station with RFID card and password panel when the user input true password door is opened. Relay control panel provides the smart plug application.



I used Arduino IDE for the programming the stations. Firstly, Nodemcu has connected the WiFi and after that, it tries to connect MQTT server. This server IP is raspberry pi’s IP and starts to the 10000msec delay time. It means Nodemcu sends Sensor values to the raspberry pi server each 10 sec. Also, you can change this delay time by using the website or mobile app.


I use Node.js for Website, Database and MQTT Server. I design to theme with using CSS and EJS. Ejs is similar to HTML but EJS is more useful and works with Node.js. Also, we used for when values come to the server, the page does not change only value is changed so Website does not use a lot of cache memory. Also, we used Node.js Express for more functional structure for Website. Also, we visualized old data and live data with Chart.js graph


Chart.js is the best graph library for Node.js.

Also, these graphs are dynamic and live. We can use all data in the same graph. Above graph is the live data for station 1. It shows only 10 new data. You can change the display, for example, you can show only Humidity and Temperature value and

close other values showing. Also, the second figure shows old values graph using MongoDB database is one of the socket programming libraries for javascript. Usually, this is used for a chat application. However,

we use this library for live data coming from sensors. Also the changes icon in case of value that came from sensors.


These icons change in the webpage with sensors data and give the status of house’s room. For

example (cloudy, rainy, humid, sunny, dark, bright, hot, cold, etc). The above figure show Status and live values with using


We use bootstrap for design display website with using CSS and HTML. Bootstrap allows the table visualization and display settings for all kind of resolutions.


Raspberry Pi 3 Access Point

Make the Access point on Linux operating system, dnsmasq , hostapd programs are required.

And also it required some changes in the startup interfaces file.

MQTT Server

For the MQTT server, we use Mosca library for Node.js MQTT application and we use port 1883 for MQTT server. Mosca library is one of the socket programmings. Also, It is very simple.

This is the terminals of the server and wireless communication broker.


I use MongoDB for the database, MongoDB is NoSQL database, and therefore, I can save big data without a table so I can use these data fast. Above figure shows the database query. you can show the results with chart or value. Bellow figure shows query result value.


Database report shows when you search type is Report on the database. Server start to find data in the database and calculate the average for all value and if read the specific value in database push the new array this values and after that calculate the average for data when each specific situation happened. After that shows graph and value.

Analysis and Results

At the beginning of the project, my main issue is providing cloud service and network connection for all station. I encountered a few problems and difficulties while working on this issue. These are often software problems. Nodemcu can’t connect WPA-EAP network so I can’t connect to our university networks and I designed the access point on our university network. The other problem is the Nodemcu sleep system. I can turn off the Nodemcu with specific time but I can’t wake the sleep before the scheduled time. I want to wake Nodemcu with using ping but can’t find how to do it.

However, this new connection provides multiple users and improve speed and bandwidth limit.


Internet of things and Augmented Reality big areas and trending technologies. In this project, I put industry 4.0 topics together which are Augmented Reality Internet of things, Big Data, No-SQL database, Machine to Machine connection, Cross platforms and Cloud systems. Beginning to use them together will provide great fun and convenience to the user. However, in today there are few examples of these technologies used together. This project will be motivation for next generation. Also, this project is more cheapie than any other applications.














One thought on “IoT-AR | Augmented Reality and Internet of Things; Smart Home

Leave a Reply

Your email address will not be published. Required fields are marked *