Introduction
Dev Technology entered and won the GSA’s AI/ML Technical Challenge for automating the rejection of Enterprise User License Agreement (EULA) clauses which are unacceptable to US Federal contractual standards. We competed against some 60 commercial, academic and individual entrants. Our solution used Deep NLP to achieve a 91% accuracy based on only 8000 example clauses. Our solution also explained its determination of acceptability by showing users examples of established acceptable and unacceptable clauses. Our solution was packaged as a machine-learning enabled API and a single-page web application, and can process EULA texts in commonly used formats.
Regulatory and legal acceptability of EULA Clauses is a ‘zero-tolerance’ style determination – no matter how much acceptable text may exist within a clause, if one part of it is identifiably unacceptable, the whole clause is considered unacceptable. This differs from traditional sentiment analysis or topical classification. Dev Technology’s solution breaks clauses into text ‘windows’ and decides if a window contains unacceptable language which can poison the whole clause.
In training our classification model, our team employed transfer learning from the Bidirectional Encoder Representations from Transformers (BERT). BERT is an open source library which uses deep learning to encode linguistic knowledge about tens of thousands of words from millions of documents. These encodings capture multiple contextual meanings and provide an excellent transfer learning basis to address NLP tasks which may otherwise suffer from small training set sizes.
Automation of Clause Rejection
Dev Technology’s AI model reduced the time EULA clause analysis takes from hours to seconds. The two primary metrics for text classification are precision – how often are the clauses the model identifies as unacceptable really so; and recall – how often does the model catch the unacceptable clauses which are present in a EULA. The higher the precision of a model, the less often a human analyst must reverse an automated decision. The higher the recall of a model, the more confident a human can be that all the unacceptable clauses were found. Dev Technology’s model scored over 90% in both metrics, for a combined 91% overall accuracy score.
Explanatory AI
High-quality AI classifiers not only achieve a good accuracy rate, but can also explain their decisions using human-understandable data. Dev Technology’s EULA clause classifier provides evidence of known acceptable and unacceptable clauses from standard reference materials which most closely match the clause it made its decision about. This similarity calculation is performed at classification time, and stored as metadata for use in a user-interface or batch dataset creation.
AI Model Delivery
The EULA clause classifier was developed using traditional data science tools such as open source Python machine learning libraries, the large BERT NLP model and Jupyter Notebooks development environment. All of these tools greatly accelerate the AI/ML development process, but none of them are acceptable deliverables to an operational environment supporting business users. Dev Technology packaged the final, tuned model as a RESTful API, with an accompanying single-page web application. The clause classifier can be invoked either one document at a time, or as a batch process over many documents. The results can be visualized using HTML and JavaScript, or can be saved as a dataset compatible with all major business intelligence tools.
Overall Results
Dev Technology’s surprise win in their first entry in a Federally sponsored AI/ML technical challenge validates both their domain knowledge in AI/ML and their ability to rapidly prototype, refine and deliver advanced technology applications that support their customers’ business needs. Dev Technology meets the mission-critical needs of government agencies by exceeding our clients’ expectations through partnership, a commitment to teamwork, collaboration, and valuing our employees.
