Augmented Reality Magnification Aid For Low Vision Patients (Magnifi-AR)
Augmented reality (AR) systems that enhance visual capabilities could make text and other fine details more accessible for low vision users, improving independence and quality of life. In this project, I will make the augmented reality magnifier for low vision patients. The main purpose of Magnifi-AR is improving to low vision patients experience in visual contents (books, photos, videos). Magnifi-AR is head-mounted, hands-free wearable projection technology. This project will be ideal solution for people with Macular Degeneration. A patient will become capable of controlling content’s size, color, contrasts by using camera and projection device. The output of head-worn AR magnifier device will improve the patient‘s visual experience and comfort on different visual contents. This device will be powered by Android operating system so that more application can be added such as object & face detection applications in future.
Augmented Reality (AR) is trending technology and have the capacity to revolutionize industrial and individual use in future generations. When this technology is used by medical purposes, it is obvious that a good quality and useful concept will be created. So, I want to apply this technology to help people who is low vision patients. In this system, the user or patient will be able to control visual content easily and magnified with projector. In the project that I want to do, I will create a new approach to low vision aids it is the head mounted magnifier with projector. In this prototype, users will able to control visual contents easily and can see content where they want to see. Because head mounted magnifier will use Pico laser projector which is not necessary focus so that user can be display content without focusing problem
In the future, this project will be able to determine in advance the possible emergencies by adding in the machine learning feature, and it can draw up a new content controller with self-learning.
Firstly, I had to decide which projector and camera to use for designing head mounted display. It is important for the project’s processing. After some researching I decided to use Pico laser projector because this projector has use laser light so that when open the projector no need the focusing.
Another important thing the camera which is good for image processing and optical character recognition. After some researching, I decide use OpenCV library because it have support cross platform and open source library. I draw final concept for use in this project at Figure 2.1.1. In this project, addition camera takes picture of each frame and OpenCV try to optical character recognition which is context is text and magnify selected content and create new camera feed after that Android device mirror the screen to projector with display adaptor and user can projected magnified content wherever he/she wants. This communication example showing at Figure2.1.2.
The software part of this project will be based on the OpenCV communication protocol but in this project, I use different programming language for the different purpose. I design software architecture as appropriate to develop. Because project can be change with feedbacks from patients.
This project has 4 different components which are, raspberry pi 3, retro-reflector screen, Raspberry pi camera and pico projector. I tried to use cheap and useful components. Raspberry Pi 3 use the pico projector as a monitor. Pico-screen display content over retro-reflector screen. Therefore, user can see both outside and content.
Raspberry Pi 3
The Raspberry Pi3 is the latest version of the credit card sized computer from the Raspberry Pi Foundation. The Raspberry Pi was released by Premier Farnell, the company behind element14, in 2012. The Raspberry Pi 3 is about the 50% faster than its predecessor the Pi 2. It includes built in WIFI and Bluetooth Low Energy connectivity, making it truly an IOT-ready device.
The Raspberry Pi 3 features a 1.2 GHz quad-core 64-bit Arm Cortex A53 processor, Chip antenna, 4 USB ports, an Ethernet Port, a GPIO, HDMI, 3.5mm Audio Output, WIFI chip, 1GB LPDDR2 for RAM Memory, and a MicroSD slot. The MicroSD card contains the Pi3’s operating system and it can also be used for file storage. It also have own operating system which is Raspbian
A pico projector is a small hardware device designed to project content from a smartphone, camera, tablet, notebook or memory device onto a wall or other flat surface. Pico projectors are also known as pocket, handheld or mobile projectors and they take a number of formats that work in various ways. My pico-projector is laser based microvision pico-projector. This proejector laser-based so that, no need for focus for different distance
The Raspberry Pi Camera v2 is a high quality 8 megapixel Sony IMX219 image sensor custom designed add-on board for Raspberry Pi, featuring a fixed focus lens. It’s capable of 3280 x 2464 pixel static images, and also supports 1080p30, 720p60 and 640x480p60/90 video.
Retro-reflective screen, which transparent and can be used as head-up-display (HUD) or a see-through screen for head mounted projection displays (HMPD) is proposed. The high optical gain of screen enables the use of low power projectors to produce very bright content. The screen assembly is based on retro-reflective microspheres, patterned on an optically clear substrate using steel stencil as a shadow mask. The incident light is retro-reflected as a narrow angular cone to create an eyebox for the viewer. The optical gain and transparency of screen is varied by changing the fill factor of the mask.
For the computer vision application, I use the OpenCV libraries for the python programming language. Also I addapt the MQTT libraries for the communication mobile phone. OpenCV change the content for given mobile phone input. For example zoom in zoom out change colour and scan the paper. For scanning the paper, I use this steps;
- Take picture of the camera view
- Convert grayscale
- Apply Gassuian Blur
- Apply Canny Edge detection
- Find biggest 4 contours
- Wrapped image given contours
- Convert wrapped image to grayscale
- Convert Black and White
For mobile application I use the Unity platform with C# programming language to make the mobile application. Because Unity applications can be used cross-platform. In the mobile application each button have specific ID when it pressed, mqtt client send the ID to broker and this broker send message to Raspberry Pi.
I use MQTT library for communication devices. I decide machine-to-machine protocol. Because in the future, device number can be increase, MQTT can handle multi-device communication. I use node.js programming langage to creating broker. This broker acts like server for devices.
At the beginning of the project, my main issue is providing 30 fps in real-time and high-quality magnifying text. I encountered a few problems and difficulties while working on this issue. These are often hardware problems. Raspberry pi 3 can’t handle 30 fps in 1080p camera render. Therefore, I decrease quality in to real-time mode but when image progressing start, I increase camera image quality to take picture. Also using OpenCV with Raspberry pi camera is different than normal web cam.
As I have mentioned in the Introduction part, Augmented Reality big area and trending technology. My goal in this project is to bring together AR and medical purposes to improve and facilitate patients experience in visual content. This system is different other low visual aids projects and application. In this project can see content where he/she wants. Because projector and camera can be moveable. In this project, I put hot topics together which are Augmented Reality and Retro-reflective display technologies. This project is proof of contept for head mounted projection based augmented reality display. This project can be motivation for nex generation visual aid project. Also this project is cheaper than other visual aid project.