Abstract – This paper studies the use of facial recognition technologies to prevent crime. The most common technologies that are being used for security and authentication purposes are analyzed. The Eigenface method is the most used facial recognition technology, it can be used for security and authentication purposes. This method focuses on the aspects of the face stimuli that are important for identification, this is done by decoding face images into significant local and global ‘features’.
There are many ways for law enforcement to help them in decreasing the amount of crime. Four of these ways that use facial recognition are: FaceIt, matching faces from live security images, face recognition in photographs and face recognition from sketches. As these technologies are improving very rapidly, the dangers and ethical issues also have to be taken into account before the technologies can actually be used in our daily lives.
From this paper can be concluded that facial recognition creates a lot of opportunities to help prevent crime. However, there are still a lot of difficulties that can cause problems when these techniques are used in the real world.
1. Introduction
Every day a lot of crimes take place. Criminality is a big problem all over the world. It is a challenge to track down criminals and Artificial Intelligence could help with this. The technologies that exist nowadays make it easier to identify individuals. One of these technologies is facial recognition. These technologies have the ability to identify criminals. Facial recognition algorithms can compare two sets of data. When a match has been found, a person could be identified.
This leads to the following question: “How can facial recognition be used to prevent everyday crime?” The research question is limited to everyday crime, since facial recognition is used in a lot of fields. By everyday crime is meant: robbery, violence, drug sales, insults, threats, forgery, driving under influence, growing hemp, theft, rape, (severe) abuse and murder. To answer the main research question a few subjects will be reviewed. The facial recognition technologies which exists and how they work will be discussed. The results of what the technologies have brought so far will be evaluated. But also the downsides, risks and ethical issues of facial recognition technologies will be taken into account in this paper. Furthermore the privacy law will be discussed in this paper, with focus on the European law. Taken all these things into consideration, there can be given an answer to our main research question.
2. Facial recognition technologies
i. Which facial recognition technologies exist?
A numerous of facial recognition techniques and methods are being used for security and authentication purposes which includes areas in detective agencies and military purposes[25]. This way, the facial recognition techniques can play a role in preventing crime. There are multiple methods considered the two primary tasks of facial recognition, i.e. verification and identification. Verification, also called authentication, is presenting a face image of an unknown individual along with a claim of identity, and then ascertaining whether the individual is who he/she claims to be. Identification, also called recognition, is presenting an image of an unknown individual and determining that person’s identity by comparing that image with a database of images of known individuals[15]. There are even some facial recognition techniques that are able to see emotions. These techniques, as well as verification and identification are of great importance when it comes to finding criminals so crime can be prevented.
A few examples of the many methods and algorithm that can be used within the field of facial recognition are: Geometric Feature Based Methods, Template Based Methods, Correlation Based Methods, Matching, Pursuit Based Methods, Singular Value Decomposition Based Methods, The Dynamic Link, Matching Methods, Illumination Invariant Processing Methods, Support Vector Machine Approach, Karhunen- Loeve Expansion Based Methods, Feature Based Methods, Neural Network Based Algorithms and Model Based Methods [25].
Later on, one of the most known methods will be discussed in a detailed way. The facial recognition methods that can be used, all have a different approach. Some are more frequently used for facial recognition algorithms than others. The use of a method also depends on the needed applications. For instance, surveillance applications may best be served by capturing face images by means of a video camera while image database investigations may require static intensity images taken by a standard camera. Some other applications, such as access to top security domains, may even necessitate the forgoing of the nonintrusive quality of face recognition by requiring the user to stand in front of a 3D scanner or an infrared sensor[15]. Consequently, there can be concluded that there can be made a division of three groups of face recognition techniques, depending on the wanted type of data results, i.e. methods that compare images, methods that look at data from video cameras and methods that deal with other sensory data, like 3D pictures or infrared imagery. All of them can be used in different ways, to prevent crime from happening or recurring.
ii. How do these technologies work?
As listed above, there exists a long list of methods and algorithms that can be used for facial recognition. Four of them are used frequently and are most known in the literature, i.e. Eigenface Method, Correlation Method, Fisherface Method and the Linear Subspaces Method. But how do these facial recognition work? Because of word limitations, only one of those four facial recognition techniques, i.e The Eigenface Method, will be discussed. Hopefully this will give an general idea of how facial recognition works and can be used.
One of the major difficulties of facial recognition, is that you have to cope with the fact that a person’s appearance may change, such that the two images that are being compared differentiate too much from each other. Also environmental changes in pictures, like lightning, have to be taken into account, in order to have successful facial recognition. Thus from a picture of a face, as well as from a live face, some yet more abstract visual representation must be established which can mediate recognition despite the fact that in real life the same face will hardly ever form an identical image on successive occasions. Our ability to do this shows that we can derive structural codes for faces, which capture those aspects of the structure of a face essential to distinguish it from other faces[6].
One of the four most famous facial recognition methods is the Eigenface Method. This method focuses on the aspects of the face stimulus that are important for identification. This is done by decoding face images into significant local and global ‘features’[24]. Such features may or may not be directly related to our intuitive notion of face features such as the eyes, nose, lips and hair. Scientists Matthew Turk and Alex Pentland [24] developed a computer system for the eigenface approach which works as following: “In the language of information theory, we want to extract the relevant information in a face image, encode it as efficiently as possible, and compare one face encoding with a database of models encoded similarly.”[24]
This all happens in the following initialization operations:
1) Acquire an initial set of face images, also called the training set.
Figure 1: Images of the training set [26]
Figure 2: Eigenfaces of the training set [26]
2) Calculate the eigenfaces from the training set, keeping only the M images that correspond to the highest eigenvalues. These M images define the face space. As new faces are experienced, the eigenfaces can be updated or recalculated.
3) Calculate the corresponding distribution in M-dimensional weight space for each known individual, by projecting their face images onto the ‘face space’.
After the initialization operations, there are carried out more operations in order to recognize new face images.
4) Calculate a set of weights based on the input image and the M eigenfaces by projecting the input image onto each of the eigenfaces.
5) Determine if the image is a face at all by checking to see if the image is sufficiently close to ‘face space’.
6) If it is a face, classify the weight pattern as either a known person or as unknown.
7) (Optional) Update the eigenfaces and/or weight patterns.
8) (Optional) If the same unknown face is seen several times, calculate its characteristic weight pattern and incorporate into the known faces[24].
As mentioned earlier, there is a long list of methods that can be used for facial recognition. Four of them, i.e Eigenface Method, Correlation Method, Fisherface Method and the Linear Subspaces Method, are the most favorite. Below here, you can find the error rates of those four methods, considered pictures with close crop or the whole face.
Figure 3: Graph and table of the result of an experiment with the four most used facial recognition techniques [1]
As you can see, the Eigenface Method has the most errors, and the Fisherface Method the least. You can also see that the error rate is higher with images of close crops faces, compared to a full face image. This shows that it is harder for a facial recognition algorithm to recognize someone if their face is not fully shown in the picture and the features are thus not recognized. It also reminds us of the fact that facial recognition techniques are not completely accurate. Hopefully they will become more accurate in the future, so e.g crime can be prevented faster and better.
3. Results of face recognition technologies in crime prevention
There are many ways for law enforcement to help them with decreasing the amount of crime. Face recognition has a big role in human life. Witnesses can describe a person’s face. Also, the citizenry can help by sending in photos. But an Artificial Intelligence can also do face recognition. It can search through a database full of mug shots to find a match with the face from an image or sketch. The Artificial Intelligence will take much shorter time to find a match than a group of officers or even specialists. In the following part, there will be discussed how four different approaches are all playing a role in crime prevention.
i. FaceIt
In the department of law enforcement, there is FaceIt, a face recognition system that can search through a whole crowd for a face and match this face with the mugshot history of this specific person. FaceIt is considered to be the most accurate facial recognition software as of today [11]. This face recognition system is mostly used by law enforcement agencies. Next to being used for law enforcement and security surveillance, which is the primary use of it, the system can also be used for computer security and eliminating fraud. Eliminating fraud is very useful in a situation as voting in a presidential election. By using this technology, voting multiple times can be evaded and people can’t get away with it anymore. People’s faces will be stored in a database and if a face appears again, then that vote won’t count [2].
ii. Matching faces from live security images
The law enforcement is always on the hunt for terrorists or big criminals. Most already have good mug-shot which can help to find a face in a crowded place such as an airport [16]. And when agents are trying to hunt some criminals down, then they will probably try to flee the country. At the airport, face recognition can be of big help. The faces of these criminals are collected and stored in a database with faces of people who are not allowed to board a plane. The faces of travelers are being scanned and are being compared with the database at the security check. If the program founds a match, the guards of the airport will go to them and cameras will be turned to the person for live images. After picking the person up, the guards will try to decide whether to keep the person from going to an airplane or they decide that it is not the face that they are searching for [27].
iii. Face recognition in photographs
To search for a specific name by only having a picture of someone, you need face recognition. Not only the society can drag a picture in the uploading square of a software or site to search through a database full of different faces with matching names, but also the police has such a kind of software. As stated earlier, it occurs often that the police state a call for help from the surrounding people. They want the people to be an extra pair of eyes to tip them off if they encounter a wanted person (for example see them across the street on the telephone), who can be in hiding or not. Society can then send in pictures of the person. Before going to hunt that specific person down, the police force needs to check if this is the person they are searching for. First, they need a picture with a clear face. The police can get a couple of those pictures and they all go through a software that tries to match the face on the picture to the wanted face. If they match, then an alarm will go off that they have found the criminal and from there on they can take action. If the result is negative, then the force will wait for more images. The effectiveness of this depends on the quality of the picture [14]. If the quality of the picture, which has a match, is high, then the chance of having a “false positive” will be smaller than a picture with low quality.
But there are some disadvantages to this method. What if the police database doesn’t contain a mugshot of this specific person and the security images are not that clear, how can they match the face from the picture? This a fundamental problem in face recognition if there were no witnesses [12]. But if there were witnesses, they can match the face with a sketch. And that process can also be reversed.
iv. Face recognition in sketches
When a face cannot be obtained from a picture or from security images, a forensic sketch can be a good substitute based on the description a witness is giving. [23] These sketches include facial features and depending on those facial features, the forensic sketch can correspond to a mugshot in the police’s database full of mugshots. To see if a face matches with a mugshot from the database, face recognition is used. Again, the efficiency of this method depends on how good the quality of the sketch is. In the database there are many mugshots, so to increase the chance that the sketch will match with the good mugshot, a law enforcement agency will have to look at the best-retrieved results. They can look for example at the best 30 or 50. If the police force will only look at the best result, then the chance to find the mugshot that they want will be smaller than if they look at the best 30 or 50 results. Beside the quality, the computer can make mistakes and that is also a reason why looking at the best 30 or 50 is better than just looking at the best result of the computer [20].
4. Dangers of using facial recognition techniques
Using facial recognition in the real world is becoming more normal, and the technologies are improving very rapidly. It is just a matter of time until facial recognition is something people will use in their daily life. Until this is the case, there are still a lot of things to improve and the dangers that this technique bring need to be considered, and solutions need to be found. In this paper, a few of these dangers will be discussed and analyzed.
i. Crime opportunities will increase
When one is using facial recognition to prevent crime, it will also create opportunities for criminals. Stalking and identity fraud, for instance, will be made much easier. Cameras with facial recognition technology can be used to keep track of people. These cameras do not have to be expensive, which will make it more tempting for criminals to buy them, and use them for the wrong purposes. The facial recognition technologies can provide stalkers with more data about their victims: they can be used to get to know a person’s weekly schedule or to keep track of when someone is going away. It provides critical information, which for instance tells a criminal if it is safe for them to break into a house.
These cameras can also be used to replace the user ID/password authentication method to access computer systems to obtain services in the name of another person. Even though the new methods can effectively distinguish the real face from fake photos by calculating the depth of the face, it is not that hard to break into a system that uses facial recognition. [3][8] US senator Al Franken has given his opinion on the problem of this topic in an open letter to the creators of an app that uses facial recognition (i.e. NameTag): “Unlike other biometric identifiers such as iris scans and fingerprints, facial recognition is designed to operate at a distance, without the knowledge or consent of the person being identified,” he wrote. “Individuals cannot reasonably prevent themselves from being identified by cameras that could be anywhere – on a lamp post, attached to an unmanned aerial vehicle or, now, integrated into the eyewear of a stranger.”. [9]
ii. Racial/ethnic bias
Recent research suggests that the algorithms behind facial-recognition technology may suffer from a racial or ethnic bias: many algorithms expose differences in accuracy across race, gender and other demographics [10].
It is shown in a study by P. J. Phillips [22] that algorithms developed in East Asia recognized Asian faces far more accurately than Caucasian faces. The exact opposite was true for algorithms developed in Europe and the United states. This implies that the conditions in which an algorithm is created can influence the accuracy of its results. A possible explanation for this is that the developer of an algorithm may program it to focus on facial appearances that are more easily distinguishable in some races than in others [10][22].
It is not only in the way the algorithm is programmed. It is also in the way the algorithm is trained. It is possible that a certain algorithm has more experience with Asian faces than with Caucasian faces. This unfair representation of the population which the algorithm might me used on, will lead to problems. If you do not include many images from one ethnic subgroup, it won’t perform too well on those groups because Artificial Intelligence learns from the examples it was trained on [19][22].
In conclusion, the performance of face recognition algorithms suffers from a racial or ethnic bias. The demographic origin of the algorithm, and the demographic structure of the test population has a big influence on the accuracy of the results of the algorithm. This bias is particularly unsettling in the context of the vast racial disparities that already exist in the arrest rates [22][10].
iii. System still needs a human judge
The last problem that will be discussed in this paper is that the technologies that are existing today are far from perfect. Right now, companies are advertising their technologies as “a highly efficient and accurate tool with an identification rate above 95 percent.” (said by Facefirst.) In reality, these claims are almost impossible to verify. The facial-recognition algorithms used by police are not enforced to go through public or independent testing to determine accuracy or check for bias before being deployed on everyday citizens. This means that the companies that are making these claims, can easily revise their results, and change them if they are not high enough [9].
And even if these claims are true, an identification rate of 95 percent is not enough for any system to rely on for society. If a facial recognition system makes a decision (e.g. if a person has committed a crime, by matching the face to e.g. images collected from security cameras), the outcome is purely based on the face features of that specific person. When this same task is given to a human being, the human will base his/her decision on other factors as well (e.g. voice, height, body language, confidence), this makes the decision more authentic. Hence, to make the chances of falsely identifying a person as low as possible, the system will still need a human judge.
4. Ethics
The improvement of facial recognition technologies can be a great help for national security. But it raises strong concerns regarding the individual’s privacy. When there are cameras everywhere, individuals will be continually watched. This possible violation of privacy creates fear in people. Another point that raises fear, is that the data of facial recognition can be misused. There is always tension between the need for privacy and what this loss of privacy can bring us (security). Is it acceptable to use facial recognition technologies in all situations? That is what will be discussed in this paragraph.
A reason why facial recognition technologies should be used is because of its contribution to security. There is belief that these technologies can offer solutions, it can for example help by the identification of suspects and thus help prevent crimes. With these systems there is a broader intelligence and security infrastructure [13]. But are the technologies safe enough? There is one big ethical issue, that is the right to privacy. Misuse of data and the problem of error are related to this.
i. Privacy
Every person has the right to have privacy and protection of his or her personal data. Facial recognition systems are a threat to the individual’s privacy. It can lead to a society in which the government knows everything about an individual.
The law should protect individuals against the dangers facial recognition technologies entail. The privacy law in Europe, the GDPR, is as follows: “the use of biometric data for identification is in principle prohibited (Art. 9.1). Article 9.2 GDPR however contains many exceptions to the prohibition of bio-metric data use for identification, including the explicit consent of the person.” [18]. The use of facial recognition data is forbidden, unless a legal justification exists.
Whether this law protects persons adequately from violation of their privacy is arguable. The law allows the usage of different technologies, provided that certain principles are being followed in the application of the technology. For example, the facial recognition technology may only be used for a specific purpose and only with consent of the individual. Interpretation of the law remains difficult, especially if there is an increase of new technologies. Since the law does not apply to every specific technology there is a chance that this technology can become a risk, because some technologies can violate the privacy of individuals.
ii. Misuse of personal data
Law enforcement agencies can use facial recognition technologies to identify criminals and criminal activity [21]. But there is no guarantee that the images and identities are safe. The data can be misused and when the data is shared, the individual’s privacy is gone. The misuse of the technology is one ethical aspect that needs to be concerned. The technology may gain unintended purposes. There have been numerous incidents in which video cameras were focused on inappropriate areas, for example on bedroom windows. Once it is determined that an individual’s face does not match with the database, the facial signature is supposed to be deleted. But there is no evidence that the identities are deleted. They can also be stockpiled, so they can be used in the future and can be shared among for instance other governmental agencies [21].
iii. Problem of error
The last ethical aspect to consider is the problem of error, which is the fact that incorrect matches can be made with face recognition technologies. This can lead to accusing innocent citizens. This is not always the fault of the technologies, but can occur in any database system with personal data. It would be acceptable to use facial recognition if a good ratio can be attained between false and true positive results. Provided that the individuals who are rated as a false positive are treated well and they are not questioned in an improper way. People have accepted that they sometimes have to endure minor inconveniences to be able to detect criminals. But in case there are too many false positives for each true positive, the harm done to innocent people is more important than detecting or arresting a criminal [5]. The systems should be tested on their performance before they are used. With the information this provides, improvements can be made to the systems in order to reduce the rate of false positives [4].
The challenge is how to trade-off between privacy and the security of facial recognition systems. One cannot prevaricate the loss of privacy by using facial recognition in everyday life. But the question is, is the improvement in security enough relative to the loss of the individual’s privacy? There is not one specific answer to this question. There are and there always will be proponents and opponents for the use of facial recognition systems. The proponents state that the systems gain enough security relative to the loss of privacy. It can help to prevent crime and detect criminals, thus people feel safer. But the opponents state that there will be too much loss of privacy, the technology can make errors and can be misused. This makes people fearful.
There should be made strict guidelines concerning privacy when using facial recognition to prevent everyday crime. Furthermore, the technologies should be checked very well, so that there are no chances on false positives.
5. Conclusion
Facial recognition systems can be used in various ways to prevent crimes. One of these ways is using FaceIt. FaceIt is a face recognition system that can search through a crowd for a face and match this face with the mugshot history of this specific person. It is considered to be the most accurate facial recognition software as of today. Other results are: matching faces from live security images, face recognition in photographs and face recognition from sketches. These are all methods in which a face (of e.g. a possible criminal) is compared to a face in respectively live security images, photographs and sketches, to see if there is a match.
The facial recognition system is far from perfect. There are still a lot of dangers that need to be considered. Some of the risks that facial recognition brings are: the increasing of crime opportunities, the possibility of falsely accusing innocent people and a loss of privacy.
From this paper can be concluded that facial recognition creates a lot of opportunities to help prevent crime. However, there are still a lot of difficulties that can cause problem when these techniques are used in the real world.
Further research
Further research can be done in different ways. There are still a lot of ethical questions that need to be answered. A trade-off between privacy and the security of facial recognition systems needs to be found. Further research in this ethical field is necessary to understand the impact implementing facial recognition in the prevention of everyday criminality has.
To really understand facial recognition, the different methods and algorithms that can be used for facial recognition should be analyzed and compared with each other. This way, it should be easier to get an image of what facial recognition is and in what ways it can help with preventing crime.