When New Technology Brings New Risk

Machine learning is here to stay, so how do we assess its risk?

An article review.

When it comes to assessing technology risk, there seems to be as many methods as there are attack vectors… but what happens when an entirely new field opens up?  When it comes to machine learning (ML) there aren’t many methodologies available, but a team of Microsoft employees are working on changing that.  The team in question is working on Counterfeit, a tool for assessing the security of ML systems, and in addition to creating this command-line tool they have written some interesting guidance on how you can begin to assess ML risk on your own.

The guidance includes a brief description of the ML development lifecycle, which begins with collecting the data which will be used to train the model and continues through processing that data, feeding it to the ML algorithm, validating its output and finally putting the system into production.  Once the system is in production the job isn’t over, though: ML algorithms must be continuously monitored to ensure their output remains valid.

In addition to needing to ensure the ML system is producing valid output, a number of attacks exist that could reveal the data the model was trained with.  If the model in question was trained with personally identifiable information (PII), this could raise questions regarding compliance with standards such as HIPPA or GLBA.

Finally, the guidance looks at the methodology behind potential ML attacks.  Starting with information gathering about the application, an attacker may then focus on attacking the ML algorithm itself, either to reveal data or to repurpose the algorithm to perform a potentially malicious task.  It’s also important to realize that while ML is new, it is still a component of your existing technology infrastructure.  Therefore all the traditional technology risks are also present, and a compromised workstation or server could also lead to a potential compromise of the ML system.

Machine learning offers tantalizing potential benefits, but as it is a rapidly developing technology one must also stay ahead of the curve when it comes to new threats.  Luckily frameworks like the one provided by the Azure team are being developed to assess those risks, and it certainly won’t be the last.

Original article by the Microsoft Azure Team.