Projects:2016s1-197 Sound Triangulation for Invisible Keyboards

Project Instruction

With the rapid innovation of Human Interface Devices (HID), a common goal is to make more benefits, such as to be more portable, energy-saving and secure [1]. And the keyboard is the one of the most important HID, which is used universally in electronic devices. Therefore, improving the keyboard becomes significant. This project aims to design and implement an “invisible keyboard” by using three microphones to locate the position of keys based on the sound trilateration principle. There are three main parts to our project in terms of proposed method, system design and project management. The overall structure of system is shown in figure 1. Firstly, the sound source will be obtained by three identical microphones and then go through amplifiers respectively to amplify sound signals for three channels, which is to ensure the signals can be recognized by the PIC. After using three pin of PIC to receipt three channel signals, the PIC is able to process data, which can be transferred to the computer by a USB connector. Finally, the buttons pressed by users will display on the computer screen by using serial monitor.

Objectives

This project aims to achieve an “invisible keyboard” by using three sensors (electret microphones) to locate the position of keys (sound sources) based on the speed of sound through the air according to sound trilateration principle.

Motivation and Significance

The keyboard designed by our project group is a flat plate engraved with keys surrounded by three microphones. It does not require many materials so it is lighter and more energy-saving than a normal keyboard. In addition, a serious security risk is possible by finding the sequence of pressed keys by using normal mechanical keyboards, which does not exist in the use of keyboard in our project. Therefore, the keyboard devised by applying sound trilateration principle to locate positions of key is more portable, energy-saving and secure than a normal keyboard, which can strengthen market competitiveness of this keyboard. This report is intended to draw attention to keyboard designers, suppliers and developers of HID as well as people who are interested in HID.

Theory

2.1Trilateration Theory

Trilateration is a method to be used to locate the absolute or relative positions of points by measuring the distances and using the geometry of circles, spheres or triangles [2]. Compared with triangulation, it does not measure angles, but distances. More specifically, trilateration applies the known positions of two or more reference points to gauge the distance between a subject and each reference point [2]. These reference points are anchored and known locations. As to a two-dimensional (2D) plane shown in figure 2, at least three reference points are required to determine the position of a subject accurately and uniquely, which are (Xi, Yi), (Xj, Yj) and (Xk, Yk). The point of a subject is (Xu, Yu). Therefore, three distances between the subject and each reference point can be measured as Diu, Dju and Dku.

Time Difference of Arrival (TDOA)

Time difference of arrival (TDOA) is the propagation delay of a signal relative to propagation time of another signal [3]. The idea of TDOA is to measure the time differences of propagation in order to locate the position of a subject point [3]. According to the trilateration theory in section 2.1, three distances (Diu, Dju and Dku) can be obtained. Correspondingly, three times (Tiu, Tju and Tku) can be calculated by Diu/ V, Dju/ V and Dku / V respectively, where V is the speed of transmission from the subject point to the reference point. As a result, there are two effective TDOA values (Δt1 and Δt2) obtained by Eq.1 and Eq. 2.

                                    Δt1 = Tju – Tiu = Dju / V – Diu / V       	                             (1)
                                    Δt2 = Tku – Tiu = Dku / V – Diu / V	                                     (2)

For our project case as the figure 3 shown, the minimum number of microphones is three according to the sound trilateration principle in two-dimensional geometry, and the reference points are the location of microphones; the subject point is the sound source. The t1 is the propagation time spent from sound source to Microphone 1; t2 is the propagation time spent from sound source to Microphone 2. Therefore, a TDOA can be acquired as Δt1 (Δt1 = t2- t1). Likewise, another TDOA can be calculated by subtracting times from sound source to Microphone 1 and Microphone 3 (Δt2 = t3- t1). It is not difficult to find that there are many points with the same TDOA (Δt1) on a curve as shown in figure 3, which is same as a hyperbola [3]. Another curve can be drawn by another TDOA (Δt2), which is also a hyperbola. The intersection of the two different hyperbolas provides a unique position of subject point (sound source) [3]. Consequently, two different TDOA can position an exclusive location in 2D case.

Proposed Approaches

According to the theory explained in section 2, obtaining two TDOA as a coordinate such as (Δt1, Δt2) is a core idea to locate a sound source. There are two feasible methods to achieve TDOA data as presented below.

Capturing entire sound signals

The first method is to capture the entire sound signals for three microphones by using analog input of PIC, and transfer the data to Matlab software, which can plot figure and detect the peak of the initial pressure pulse for each sound signal in time domain. Afterwards find the time corresponding to the peak for each one. As a result, the TDOA can be calculated by subtracting the two times, namely, t2-t1, as shown in figure 4. This method is able to track and inspect the data of sound signals received by PIC. However, using this idea has a couple of weaknesses, which cannot achieve a desired deliverable. Firstly, the speed of the keystrokes has been limited, because it is time-consuming to capture and transfer the entire sound signals for three channels, and the time of that is much longer than the time of one keystroke. Secondly, this idea is not able to achieve real-time monitoring of keystrokes, because analog reading and transferring data to Matlab are asynchronous communication. In addition, transferring data to Matlab can extend the time of analog reading indirectly. As a result, transferring data to Matlab must be at the end of analog reading of entire sound signals for three channels.

Timer

The second method is to achieve two TDOA by using a timer, which is in the PIC and used to record propagation time spent from sound source to microphones. The following is specific operations: • First sound signal arrival ---- to record timer t1 • Second sound signal arrival ---- to record the time t2 • Third sound signal arrival ---- to record the time t3 • To calculate two TDOA ---- Δt1 = t2 - t1; Δt2 = t3 – t2 Using this idea has several advantages. Firstly, there is no need to capture all data of sound signals, because it only detects the beginning of each sound signal. Secondly, using this method can reduce the factors that can influence on system and results, such as reflections or other disturbances, because the operating time is an instantaneous. In addition, this idea allows system to achieve real-time monitoring of keystrokes, which is what our project is eager to achieve. A problem requires to be considered for this method, which is how to verify the reliability of the method. For example, how to ensure that PIC do detect each sound signal effectively especially when the experimental results are not correct in the process of project. Therefore, this issue needs take into account in the process of software design.

Final scheme

Based on the analysis in section 3.1 and 3.2, our project group is inclined to apply the second method which is using a timer in PIC to record time, thereby obtaining TDOA. Meanwhile, Matlab can be used to plot the three sound signals that PIC receives on a graph to prove and check the validity of the data.

Hardware System

The hardware system consists of four main parts, namely, a keyboard, three microphones, three amplifiers and a PIC as figure 5 shown. Taping key engraved on keyboard will make a sound, which is picked by three fixed and identical microphones. Afterwards, the outputs of three microphones go though amplifiers respectively to amplify sound signals. Finally, PIC will receive signals and process data, and the results are shown on the screen by using serial monitor.

Keyboard

A flat plate is used as a keyboard engraved 12 “buttons” as shown in figure 6, which can be made by plastic or wood and both of them can be tapped to make a loud sound. The size of button is 1cm x 1cm, because if the button is too large, it will not save space and will trigger a big range of TDOA for each key, which may lead to trouble to distinguish the adjacent button. If the button is too small, it will be hard for users to tap button even though it reduces the difficulty for locating. Considering the points mentioned above, the 1cm x 1cm key can be an appropriate choice. The spacing between buttons is 2cm, which decides that the time of sampling to each sound signal must be less than 57.74 microseconds (us) according to the calculation in section 4.2.1. That will be a challenge for PIC. The three microphones will be placed and mounted at the edges as shown below, whose specification and design is described in the next section.

Topic

Sound Trilateration for Invisible Keyboards

Team members

Salim Al Arafati, Mingyue Xie

Supervisors

Dr Withawat Withayachumnankul, Dr. Brian Ng