“We proved-out that the technology works phenomenally well. The technology is great.” This effusive praise by an LP executive who oversaw a several-store test of facial recognition technology is perhaps most noteworthy for the retailer’s ultimate decision against implementation.
More than ever, retailers have the ability to shift security upstream—to move from reactive strategies of catching or investigating criminals to identifying them before they ever have a chance to strike. But even if technology is now ready for retailers, are retailers ready for the technology?
LP expert and consultant Walter Palmer CFI, CFE, is among those who see both potential and problems for facial recognition technology in retail. “Very few retailers are using it currently, but manufacturers are promoting it, and there are strong advocates among some LP leaders,” said Palmer. “And I think it’s clear why—the use case is brilliant. Truly. But the question is if we can get there. There are a lot of head winds.”
Facial recognition technology (FRT) and other identification technologies (see sidebar) clearly have momentum. It has captivated mainstream press attention, has overcome multiple technological hurdles, and has largely addressed the mismatch between promise and performance that held it back in years’ past. All this is leading to some significantly optimistic predictions for FRT.
A Bright Future
“FRT is fairly new for most folks. But we’re getting lots of phone calls—from just about everyone you can imagine asking about what it does, how hard it is to roll out, so we’re out there doing a lot of demos and pilots,” explained Peter Trepp, CEO of FaceFirst, which launched Sentinel-IQ, a facial recognition platform for retail security, in June. “My perspective is that well over half of retailers are at least starting to take a look. They’re beginning to understand the power that it has and learning that they’re not going to be first to use it—because nobody wants to be first. But it’s coming, and I see it accelerating.”
Charles Fleischman, COO at DeepCam, an artificial intelligence company, said he’s seeing similar momentum. “We finished training at one client, and the head guy kept saying, ‘Wow. How fast can you do another 100 stores?’ That’s where we’re going very quickly.”
It’s probably not just vendors’ wishful thinking. Driven by improved accuracy, speed, and an increased need for enhanced surveillance and monitoring, the global facial recognition market is predicted to grow from $3.85 billion in 2017 to reach $9.78 billion by 2023. “Various applications like homeland security, criminal investigation, ID management, and physical security have made the demand for this technology increase,” according to a market analysis report, Global Facial Recognition Market, 2017-2023. “The market has been growing at a tremendous rate, and the awareness on the benefits of using this technology is increasing,” it notes. Among the various processes like 2D, 3D, or facial analytics, the 3D market holds the most significant share “due to its highly and most accurate face recognitions.”
High-profile installations are likely to feed interest in FRT. In August, for example, the executive director of security for the 2020 Olympics said the Games will be using a facial recognition system by NEC Corp. to monitor and speed checkpoint processing of accredited individuals, such as staff, athletes, and media. Madison Square Garden has already used FRT at events to compare individuals against a database of known security risks, according to a report in the New York Times.
Momentum is particularly strong in the financial sector and at airports and stadiums, according to Trepp. Retail is not leading the way, but he thinks it’s ahead of most sectors. It’s also big overseas. Property developer China Overseas Land & Investment and JD.com recently announced plans to launch several hundred unmanned convenience stores throughout China that will incorporate facial recognition. In a survey by Opinion Matters of representatives from 150 retail stores in the UK, more than one-quarter said they have used facial recognition for security and/or sales and marketing.
Manufacturers are certainly recognizing the potential. For example, Megvii, a Chinese startup that develops facial recognition technology, is raising $600 million in funding, most of which reportedly will be used on the company’s initiative to sell more of its technology to retail stores.
The need for retailers to understand customers at the individual level—not just by ZIP code, age, or gender—is also likely to fuel interest. A principal analyst at Forrester Research identified that goal as a top technology trend at the 2018 NRFtech conference.
Lofty expectations are the result of significant technology enhancements that manufacturers have made. “A lot has happened in the last five years—and even more so in the last two or three years—especially in terms of a big leap forward in accuracy and performance,” said Trepp. “We’ve now gotten to the level that really was sort of the promise all along, that ability to match a person who moves in front of a camera in a real-time environment.” By identifying a face against those contained in a database, and doing so in real time, systems can send an immediate alert to store personnel, which allows LP to monitor or intervene with a suspicious person before a crime is committed. “That has been the big leap,” said Trepp.
The enhanced capability is made possible by the shift to neural networks for feature matching and a continued improvement in computing power, according to Don Knasel, CEO and founder of DeepCam, which launched Retail by DeepCam at ISC West in 2018. “All world-class facial recognition is now based on neural networks,” he said. It enhances the value of systems in multiple ways, including being able to make comparisons of partial faces, when faces are captured at an angle, and has improved accuracy across demographics and despite beards, sunglasses, and hats. Also, it enhances the ability to run recognition on less expensive cameras, said Knasel, “which brings the cost of a system down by an order of magnitude.”
The FRT market study made note of the progress manufacturers have made, concluding that traditional technology constraints regarding camera type and lighting conditions have largely been overcome and predicting that improvements haven’t yet hit a plateau. “A lot of research is being done to make the technology more efficient, and huge new players are competing in the market to develop better and more-efficient systems,” according to the study. Moreover, some experts noted that since recognition is based on a fundamental human feature, rather than a type of device they interact with, FRT’s life cycle could be especially long.
It’s still a rarity in US retail stores, but some are already having success identifying chronic returners, counterfeiters, thieves, and other persons of interest-and it’s creating interest among others. “We’re making scores of matches every day, and at least once a week, we get a really, really bad guy,” said Trepp.
Another great use, according to Andrew Chapman, senior vice president of sales at 3VR, is to identify if an individual who comes in empty-handed is shortly thereafter at the return desk carrying a package. “Some retailers use that to at least reject the return, and for that it’s a great tool,” he said.
Better performance, lower cost, real-world uses, early adopter proof-of-concepts—it’s clear why there is a buzz about facial recognition. Joel Rieger, principal consultant for Rieger Consulting, is currently helping several major retailers examine technology platforms and conduct tests. “The willingness of retail companies to engage with this type of technology is certainly on the increase,” he told LP Magazine.
Yet, as it looks to become another standard tool in the LP arsenal, FRT is contending with some significant historic, regulatory, strategic, and public-perception obstacles that has kept many retailers on the sideline.
A Spotty History
A lingering hangover of false promises may still afflict some LP leaders’ attitudes toward FRT, contributing to their cautious approach to the technology.
“History is against us—this idea that facial recognition technology doesn’t work, it’s too expensive, it doesn’t work on some ethnic groups, that you’re going to get false positives and a false alert two out of three times,” said DeepCam’s Fleischman. He sees confidence in the technology growing but acknowledges it has taken time.
FaceFirst’s Trepp admits that five years ago FRT systems were not delivering on their hype, including for real-time accuracy. “It put people off,” he said.
Times have changed, however. False alerts were not an issue for one major retailer that ran a test of FRT in several big-box stores, according to its project lead on the test. “We had zero false positives, 100 percent accuracy. And that was great. We were hoping it would be close to that, but the system exceeded expectations. There were no false promises. That was actually the best part of that test,” he said.
Shifting Sentiments
Although retail leaders may be impressed by how the technology has progressed, concern over consumer reaction is a significant counterweight as they gauge whether to forge ahead. “Retailers are scared of the public’s perception,” Palmer believes.
According to its test project leader, media attacks on companies over privacy issues filtered into the big-box retailer’s decision not to widely implement FRT. He added that the case for the technology would be easier to make if it weren’t for a residual sense of risk from public sentiment and negative brand impact.
Moreover, when public attitudes are the issue, criticism can take a toll whether or not it’s earned. Privacy advocates generate a lot of press when they float potential horror stories associated with the collection of biometric data, so the potential of bad press and harm to a retail brand, even if it’s abstract, surely acts as a drag on FRT adoption by retailers.
In response, some vendors are taking steps to educate the public, such as using terms like “matching” instead of “recognition.” This may help promote the fact that just because a system can recognize that an individual was previously in a store does not necessarily mean the system knows who you are.
Time, however, is probably the surest answer to shoppers’ skittishness. Consumers have repeatedly shown willingness to trade privacy for convenience and broadly participate on platforms that track where they go and what they do—even after privacy snafus come to light. “The obstacle of public perception [of FRT] is something that could be overcome, I think. There is already a significant part of the population that is comfortable being tracked,” said Palmer, who recalled that initial outcry over public uses of video surveillance waned over time and then largely disappeared. The same, he thinks, could eventually happen with FRT.
The “ick” factor is likely to dissipate and may do so quickly if past consumer behavior is a good indicator. Still, it’s not gone just yet, according to data on consumer sentiment. For example, surveys in the last year have shown consumers somewhat split on their desire to purchase devices that use facial recognition to help protect their privacy. It’s easy to think that consumers, currently, might be no more favorable to retail stores using the technology for theft prevention.
Regulatory Roadblocks?
From a privacy perspective, facial feature matching and other identification technologies can operate as opt-in technologies, flagging only people who commit an offense, and even then, no personal identifying information needs to be maintained, just a picture with its own unique number and time, date, and location. “It is nothing more than you have from CCTV, and less, because that is actually video of the whole time a person is in a store,” said Fleischman.
Rieger said you can’t gain access to law enforcement databases, but that nothing prevents a retailer from creating its own. “Traditionally, LP would bust a guy, take a picture of his license, develop a case log, and take his picture. And nothing prevents LP from doing this same thing enterprise-wide with the help of technology.” He added that nothing prohibits a retailer from then using that data to identify when that person is active, in what radius, and so on. And while LP can’t access law enforcement data, it’s possible to send the data the other way, said Rieger, which could strengthen a retailer’s public-private partnerships and potentially result in the disruption of organized retail crime (ORC) operations.
As with consumer sentiment, however, general regulatory uncertainty—rather than the legitimacy of privacy concerns or current law—could be enough to give a retailer pause. “It’s standard in common law that people have no expectation of privacy in a public venue, and the public is an invitee to a store, so there is an ability to disinvite people,” said Palmer. But even if the legal footing is currently solid, recent regulation related to privacy and biometrics in a few states, like California and Illinois, and the prospect of what other states may do, can damper a national chain’s enthusiasm.
News items of biometric systems coming under attack surely don’t help, including a proposed class action in August against Crate & Barrel, claiming its use of biometric fingerprint scanners violate Illinois’ Biometric Information Privacy Act. Liquor store chain Binny’s Beverage Depot was also hit with a potential class action under the Illinois law for its fingerprint-based timekeeping system. The New York Civil Liberties Union is trying to block a school district in the western part of the state from implementing facial recognition to alert security to the presence of an expelled student, sex offenders, or disgruntled ex-employees. And, under public pressure, Orlando’s police department let its pilot program with Amazon’s Rekognition facial technology expire. The department was testing the service’s ability to aid criminal investigations by recognizing suspects in photos and videos but faced vocal criticism by civil rights groups. While such developments haven’t materially impacted the ability of retailers to make use of FRT, they’re probably not quelling concerns either.
“When you have the prospect of laws that could allow for civil damages and the right to sue as an individual, then you’re opening up a whole can of worms,” said Palmer. He said the question for a national retailer is therefore, “Do you make a leap of faith when what and how you can implement is not exactly clear? You have practitioners who like to try all this stuff and evidence it works, but with legal departments concerned with potential liability, retailers might ask the question, ‘Is the juice really worth the squeeze?'”
Decision Points
The issues above might not be deal breakers by themselves, but they may add up for some retailers to choose a wait-and-see approach. “As a whole, I think the technology could have a lot of value in targeted markets. If the cost was appropriate and the risks were minimal in terms of public perception and brand impact and changes in legislation, it definitely could have value,” noted the LP executive described at the outset. Ultimately, though, he said his company couldn’t validate the overall value of launching it across the company “and chose not to prioritize it.”
For other retailers it has been a good fit, including a major retailer profiled in LP Magazine several years ago. (See “Facial Recognition Security Cameras: A Game-Changing Technology” for an in-depth look into this retailer’s experience.) It’s also working for Jack Patel, owner of two 24/7 convenience stores on the East Coast.
Patel had been looking for technology to deploy for a while, noting that he’s always been intrigued by facial recognition and was acquainted with its use in other countries. Months after putting DeepCam’s solution in his store, he says he’s getting the results he’d hoped for.
By immediately alerting personnel to the presence of known shoplifters, via tablets at the point of sale, his stores have successfully banned problem individuals from entering. And he suspects that word has gotten around among thieves in the area, so deterrence is multiplying the system’s value. “They don’t know about the alerts, but it makes them think that different store associates know who they are and are paying very specific attention. And I’m sure they’re telling their friends, ‘Don’t go in there, because they’ll get you.'”
He says he’s also using FRT to notify staff when particularly good customers enter the store, which is particularly useful because his high turnover among store personnel doesn’t allow for this type of valuable customer service to evolve naturally. Finally, Patel says he’s also finding value from the system as both a timesaver—pointing to problems rather than aimlessly reviewing video—and a driver of better employee behavior. By reviewing video flagged as suspicious, Patel says he identifies a dozen or so people every day who stole from a store. And, for example, by identifying when employees enter and leave the store, Patel observed a problem with one worker leaving the store for extended periods during overnight hours. “This helps me to monitor them in an efficient manner,” Patel said.
Fleishman said other stores are using it for similar intelligence, from employees who take too many smoke breaks to clerks who allow friends to hang out in the store against policy. To date, Rieger says that, surprisingly, using systems for internal purposes is bigger than training them on external populations—for making sure that people in sensitive areas belong there, for validation into a stock room for a large workforce, and similar employee and contractor control.
FRT is one in a class of technologies that are offering stores new opportunities to be proactive in the fight against theft and retail crime, to transition from investigating theft to identifying suspects and preventing it from occurring, and to preidentify threats.
As plainly preferable as such a move sounds, it also contains layers of strategic implications that can muddy that seemingly compelling strategic shift. For example, while Patel is successfully utilizing alerts to ban problem individuals as they enter his convenience store, the big-box retailer’s test suggested it’s a tougher task in an environment with many entry points and where responding staff isn’t positioned at the front of the store but may, instead, be in an LP office in the back and tending to other duties.
“A lot depends on what you’re going to do [when staff gets an alert]. If you’re intent on approaching them and asking them to leave, it’s more challenging. They’d need to immediately drop what they’re doing, and by the time they make it to the front end, the person is probably well within the store. Now they need to look for the individual. And the more time that passes before they find them, the more likely they’re either not going to find them or potentially find the wrong person.” He said that complication doesn’t altogether negate the benefit of getting an alert, but it was among the components that left him with the impression that a smaller, high-risk specialty store might find it easier to realize value.
It’s probably inevitable that facial recognition will become a standard tool in the LP arsenal, taking its place alongside video, alarms, motion sensors, and officers to mitigate shrinkage and violent crime. On the road there, retailers are faced with a slew of consideration as they look at if, when, and how best to use the technology. Here are issues and advice offered up in interviews with the top vendors, LP directors who use and have tested the technology, and consultants that have helped clients with testing and implementing FRT systems:
- Price and performance. “It’s not CSI yet,” warned Rieger. “That idea that you can take a cloudy image and turn it into a crystal-clear picture…it’s better than it was five years ago, but you’re not going to generate something out of nothing. And with a wig and hat, it’s not going to recognize me.” If there is an insufficient number of points to triangulate, a system can’t generate an informed match, Rieger noted, in which case it can return multiple individuals. “Accuracy isn’t necessarily the problem; it’s more whether you’re not going to get too many false hits and distract staff. That and whether you have the capital expense funds to get the technology you need to make it work.” How much you can spend will influence the speed of the results you’re going to get, he added.
- Theft rates. Consider the current amount of store theft to calculate a system’s potential value. “When it comes to alerting if someone previously apprehended enters the building, we would have to enroll a whole lot of people to pay for one store’s system in order to realize an ROI. We simply don’t apprehend that volume of people,” said one system tester. “Certainly any retailer that has an ORC or a local loss prevention problem and high-loss stores is going to see the technology become effective faster,” said Fleischman.
- Marketing. Sweetening the business case through marketing and sales uses is an attractive option. As Patel’s real-world, VIP customer enhancement effort indicates, there are opportunities for positive uses of FRT and other identification technologies. Both Palmer and Rieger think approaching solutions from both marketing and security perspectives can make for a more compelling business case. “Positive uses are available, such as for a loyalty program. The technology can absolutely do it,” said Rieger.
- Databases. In examining a prospective system, get clarity on how data is controlled and where matching data comes from. That can be important information to consider alongside an assessment of issues associated with potential changes in privacy law.
- Bundling. Investigate the ability to tie in other analytics. Facial recognition, as a standalone, will not provide the same value to LP as bundling it with other data sources to improve its accuracy and broaden its utility, said one expert.
Uma Welingkar - Partnering. For effective applications in retail, LP leaders shouldn’t try to go it alone. “Absolutely, partnerships with IT and working with technology teams is an important part of it,” said Uma Welingkar, product manager at 3VR. It seems to be the thing that best enables projects to progress past the testing stage, she suggested.
- Maintenance. Pay attention to a system’s need for servicing during a technology trial. “It proved challenging how sensitive the equipment was. Even just the HVAC unit kicking on could blow toward a camera and throw it out of whack and need a service person to adjust it,” said one LP director involved in a FRT trial. “It was more sensitive than I expected and needed fine tuning, although I expect that would have improved over time.”
The most critical piece of advice, identified by several experts, is to review and plan the operational aspects of a system and to not just measure the technology’s ability to provide high-confidence matches. They stressed that to accurately assess a system’s potential value you need to know how you can and want to use it:
- Can you send an alert to multiple mobile devices? A central location? What does it consist of?
- Will you leverage value by assessing patterns, such as the days of the week or the time of day when events are happening? Recidivism across a network of stores?
- If you’re going to have staff approach identified suspicious persons, is training sufficient? Is turnover too high?
- What are the time constraints of the people who will be needed to manage the process? How long will it take to follow your process for validating an alert? Is it time and resources you have available?
To effectively operationalize FRT, planning, policies, and practices need to accompany the technology, warned Rieger, and setting up a pilot with a vendor to test the system and compare it to control stores is a good way to answer those questions. “It’s not a technology that you want to be afraid of,” said Rieger. As a tool against ORC, he thinks it potentially holds significant value for retailers. “But there is a lot to consider, a lot of questions to answer, and departments like HR and legal to get involved. FRT is not a technology that you can just put in.”
SIDEBAR: Not Just a Pretty Face
Facial-recognition technology is one in a class of technologies that are offering stores the chance to be more proactive in the fight against theft and retail crime.
Another opportunity to identify shoppers—but sidestep privacy concerns and use existing infrastructure—is via cameras embedded with an algorithm that picks up signals from shoppers’ WiFi- and Bluetooth-emitting devices, like smartphones, Fitbits, and Apple watches.
ClickIt’s Virtual Lineup system, for example, assigns each individual a unique but anonymous identifier, which facilitates data collection such as where that individual goes in the store, which aisles they visit, how long they shop, and—so long as they’re in possession of the same device—whenever the individual returns or visits another of the retailer’s locations. But that knowledge is never linked in a database with unique identifying information. The marketing and sales benefits are obvious, and LP has found use cases as well. For example, a criminal who covers his face during a robbery may nonetheless be easily caught by searching for other times his device was in a retailer’s location and reviewing corresponding video to see his unobscured face. Or if a known problem individual enters any of a retailer’s locations, store personnel can receive an alert.
There is also face recognition technology that is only interested in seeing a face—without concern for who it belongs to. Tied to an entry door using magnetic locks, “face detection” technology requires a clear facial image of an individual before he or she can open the door. The technology provides convenience and other stores that face a heightened risk of robbery because of late-night hours a method to reduce risk by denying entry to individuals who wear masks or otherwise attempt to conceal their identity. FKG Oil reported that it’s using First Line Facial Recognition by Blue Line Technology in select Moto Mart locations and, based on positive results, plans to expand its use. Blue Line explains that during overnight hours, store doors remain locked until a surveillance camera outside the store entrance captures a clear image of an approaching customer’s face, at which time software unlocks the door, “preventing incidents by forcing potentially violent criminals to be videotaped.”
There is also recognition technology that doesn’t look at faces. Automatic license plate recognition (ALPR) can play a role in retail’s effort to combat ORC, according to consultant Joel Rieger. He has helped major retailers implement systems that use ALPR to create white lists for entry into employee parking areas and for security trendspotting, such as recognizing a car that makes an unusual number of passbys. Canadian Tire is one retailer that reports success using ALPR. Utilizing 3VR’s VIP License Plate Recognition, one store owner says she uses the tool to verify parking lot activity as well as adherence to store policy, reviewing that each car entering a service bay has a valid service ticket.
Finally, most FRT systems enable LP to do more than recognize faces. Retail by DeepCam, for example, is designed to tackle shoplifting holistically. First, through its “advise” technology, AI points investigators to video of individuals who were in the store and exhibited suspicious behaviors. If shoplifting is identified, the system’s “match” technology permits the retailer to flag those individuals. If they again enter either that store or any other location with which the data is shared, personnel will receive an alert.
Using AI, the system is designed to get better over time at recognizing suspicious behaviors, increasing its benefit to retailers the longer they use it. As such, the ability of a technology tool that can alert a store agent when an individual in aisle 6 is about to steal something no longer seems impossible. With AI, machines can get quite skilled at recognizing when someone looks suspicious in the same way that a good LP agent has always done, said Rieger. “Matching” technology is also likely to change. “As we go forward and evolve this technology as we get the training data, it will be looking at more than the face, such as the whole body positioning or their gait, and use that to be able to say, ‘This is the same person,'” said DeepCam COO Charles Fleischman.
