AI and Machine Learning: Defense Mechanisms That Need to Be Defended

David Rios

David Rios Insua

David Rios Insua is AXA Chair in Adversarial Risk Analysis at ICMAT-CSIC and Member of the Spanish Royal Academy of Sciences. He holds the DeGroot Award from ISBA and led the Games and Decisions in Risk and Reliability program at SAMSI. He has held research and/or teaching positions at Duke, Purdue, IIASA, Aalto, Paris-Dauphine, Shanghai University for Science and Technology, CNR-IMATI, UCM, UPM and URJC.

He is a specialist in Bayesian analysis, decision analysis and risk analysis and their applications to security and cyber security. He is Scientific Director of Aisoy Robotics.

Machines are getting increasingly smarter. Cars can now help you plan your itinerary and help you park, sensing the trees, pavements and surrounding vehicles and activating the brakes as needed. In a not-so-distant future, they might routinely transport us from home to work in a driverless manner. They gather and transmit data and learn on the go, powered by artificial intelligence in a globally connected world. Are smart connected machines going to make our world more secure or, to the contrary, less so?

AI is now used by cyber security companies and governments to track down unknown vulnerabilities in their information systems and fix them before attackers exploit them. For example, automated systems can check the status of hundreds of thousands of connected devices and send warning signals to engineers when a device behaves abnormally, signalling a potential intrusion. Predictive models can also forecast imminent failures, and AI then offers precious time to react in advance.

In addition to mere scanning systems, some threat intelligence systems perform in-depth analysis of the security environment and posture within an organization. However, the entailed data deluge needs to be coherently aggregated to provide meaningful and useful risk indicators, and a combination of machine learning and economic models aid in performing such an aggregation. Threat intelligence systems can also analyze web and social network content, looking for negative online mentions of a company, which constitute a reputational threat but could also trigger cyber-attacks. This goes further than scanning, as ascertaining the nature of the tweets for example relies on advanced AI tools, such as language and sentiment analysis.

In all these cases, AI supports cyber security decision making in the presence of adversaries. New approaches, such as adversarial risk analysis, facilitate online decisions and enhance accuracy and speed in cyber risk management.

However, while the list of AI applications requiring strict security is endless (automated driving, content filters, policing and so on), AI is not immune to cyber-attacks itself. To ensure that AI applications are secure, machine learning algorithms need to be robust and reliable.

Indeed, while state-of-the-art machine learning algorithms perform extraordinarily well on standard data, they are vulnerable to so-called ‘adversarial attacks’. These attacks use data crafted precisely to fool AI. The first instance of this type of attack targeted a machine trained to recognize panda pictures. The attack led the machine to recognize a panda with high confidence when the picture was in reality, replaced by a picture of a gibbon. To achieve this, attackers simply needed to interfere during the machine learning process, presenting data that is falsely labelled — here, passing gibbons for pandas during the machine training phase. In real life, a worrying equivalent is that an autonomous car can be fooled into reading a stop sign as speed limit, and therefore not stop at the sign. Fraudsters could also disguise illegitimate insurance claims, fooling the corresponding algorithm to receive compensation. Quite importantly, attackers quickly adapt to the defense machine learning systems in place, and this could have dramatic implications in domains such as automated driving systems, defense systems, law enforcement and health to name a few.

To have artificial intelligence on board, you need to be connected. This opens many possibilities but may also leave you vulnerable.

These security issues question our standard algorithm design methods, given the presence of adaptive adversaries ready to intervene in the problem to modify the data on which we rely.

To avoid adversarial attacks, a new field called ‘adversarial machine learning’ is emerging. Its aim is to make machine learning systems robust against malicious attacks. This entails studying attacks but also defenses against attacks. For example, in spam detection, we have deployed classifying systems to detect and stop spam, but then attackers learned how to fool the protection system by changing critical words (instead of Viagra, they use VE@GR@) to make the antispam system think that a message is legitimate. We have had to learn about evolving attacks, in order to incorporate better defenses without stopping legitimate mail. The ‘adversarial machine learning’ research field uses mostly game theory to model the confrontation between learning-based systems and their adversaries.

However, in ‘adversarial machine learning’, we often assume that defenders and attackers share some information and knowledge. This assumption about sharing common knowledge is questionable in the security domain, as adversaries of course try to conceal information from each other. So we are developing another way to handle adversarial machine learning, called ‘adversarial risk analysis’, using forecasting. We model how attackers attack and react, and use this knowledge to forecast how they might attack in the future, without using the strong assumptions regarding a shared, common knowledge.

Cyber security and AI go hand in hand. As with many tools and methodologies, AI is a double-edged sword: we use modern machine learning and AI tools to design more cyber secure systems, but we need to design machine learning and AI so that they are unaffected by attacks. We need cyber security to become even more intelligent.