Mission Driven Support

Dev Technology prides itself in directly supporting our clients and their wide range of missions. We specialize in homeland security and federal law enforcement; however, we also support missions that range from the regulation of food benefits, to supporting access to seismic design data, and even preserving American history and culture.
image
image
Dev Technology AI Model Rejects 91% of Unacceptable EULA Clauses

Introduction

Dev Technology entered and won the GSA’s AI/ML Technical Challenge for automating the rejection of Enterprise User License Agreement (EULA) clauses which are unacceptable to US Federal contractual standards.  We competed against some 60 commercial, academic and individual entrants. Our solution used Deep NLP to achieve a 91% accuracy based on only 8000 example clauses. Our solution also explained its determination of acceptability by showing users examples of established acceptable and unacceptable clauses. Our solution was packaged as a machine-learning enabled API and a single-page web application, and can process EULA texts in commonly used formats.

Regulatory and legal acceptability of EULA Clauses is a ‘zero-tolerance’ style determination – no matter how much acceptable text may exist within a clause, if one part of it is identifiably unacceptable, the whole clause is considered unacceptable.  This differs from traditional sentiment analysis or topical classification.  Dev Technology’s solution breaks clauses into text ‘windows’ and decides if a window contains unacceptable language which can poison the whole clause.

In training our classification model, our team employed transfer learning from the Bidirectional Encoder Representations from Transformers (BERT).  BERT is an open source library which uses deep learning to encode linguistic knowledge about tens of thousands of words from millions of documents.  These encodings capture multiple contextual meanings and provide an excellent transfer learning basis to address NLP tasks which may otherwise suffer from small training set sizes.

Automation of Clause Rejection

Dev Technology’s AI model reduced the time EULA clause analysis takes from hours to seconds.  The two primary metrics for text classification are precision – how often are the clauses the model identifies as unacceptable really so; and recall – how often does the model catch the unacceptable clauses which are present in a EULA.  The higher the precision of a model, the less often a human analyst must reverse an automated decision.  The higher the recall of a model, the more confident a human can be that all the unacceptable clauses were found.  Dev Technology’s model scored over 90% in both metrics, for a combined 91% overall accuracy score.

Explanatory AI

High-quality AI classifiers not only achieve a good accuracy rate, but can also explain their decisions using human-understandable data.  Dev Technology’s EULA clause classifier provides evidence of known acceptable and unacceptable clauses from standard reference materials which most closely match the clause it made its decision about.  This similarity calculation is performed at classification time, and stored as metadata for use in a user-interface or batch dataset creation.

AI Model Delivery

The EULA clause classifier was developed using traditional data science tools such as open source Python machine learning libraries, the large BERT NLP model and Jupyter Notebooks development environment.  All of these tools greatly accelerate the AI/ML development process, but none of them are acceptable deliverables to an operational environment supporting business users.  Dev Technology packaged the final, tuned model as a RESTful API, with an accompanying single-page web application.  The clause classifier can be invoked either one document at a time, or as a batch process over many documents.  The results can be visualized using HTML and JavaScript, or can be saved as a dataset compatible with all major business intelligence tools.

Overall Results

Dev Technology’s surprise win in their first entry in a Federally sponsored AI/ML technical challenge validates both their domain knowledge in AI/ML and their ability to rapidly prototype, refine and deliver advanced technology applications that support their customers’ business needs.  Dev Technology meets the mission-critical needs of government agencies by exceeding our clients’ expectations through partnership, a commitment to teamwork, collaboration, and valuing our employees.

image
The Dev Technology Team develops several systems to address the backlog of processing for the Southwest Border Initiative (SWBI).

The Dev Technology Team, as part of our ICE ERO Consolidated contract worked with the ICE ERO OCIO to develop several systems to address the backlog of processing across the US Southern Border, together these systems fall under the Southwest Border Initiative (SWBI). Over the past year the team traveled to the Southwest border to sit on site, under bridges and in processing centers with ICE Officers and CBP Agents so we could see how systems were being used in the field, and to get real world feedback and suggestions from end users in high stress situations on improvements they needed in order to allow them to do their jobs faster, more accurately, and ultimately allow them to reduce the backlog of processing for non-citizens.

For the Case Acceptance System (CAS) we developed a system used by both CBP and ICE that allows CBP to request a Case Review by ICE before a non-citizen is moved from CBP Custody. The process involved a CBP Agent scanning over 40 pages of documents, attaching the newly scanned file to an email, and emailing the file to ICE Officers. ICE would then review the case files and send back information via email when the files needed to be changed and updated, before a decision could be made on a case. This process took on average 11 hours to complete. Our goal was to get that time down to under 2 hours. The team worked side by side with Agents and Officers in the field to develop a system from inception to Production in 5 weeks. The CAS Application reduced the time it took for case referrals, reviews, and decisions down to, on average, 52 minutes. With a vast majority of decisions coming in under 30 minutes. The CAS Application has now processed over 120K cases saving the end users hundreds of thousands of man hours. This allows the Agents and Officers to focus on other responsibilities.

The ATD Enrollment application was developed in very much the same fashion, the Dev team sitting with ICE Officers in soft sided facilities at the border to view firsthand the business processes and gather feedback on improvements from the end users. The ATD Enrollment application took an existing application, reengineered it with simplified workflow and connected it to an external contractor application. This application was developed from inception to production in 4.5 weeks. This gave the ICE Officers a 68%-time savings per enrollee, along with saving the contractor 50% of the time it normally took to enroll a noncitizen into the Alternate to Detention systems. The system has also cut data re-entry to zero and has drastically improved data quality across all systems. The time savings seen by the ATD Enrollment application in just the first 6 months of its deployment, at just one Southwest border site, saw a time savings of over 24,000-man hours or processing time.

The EOIR Court Scheduler Application looked to streamline the process of interacting with the Immigration Court Scheduling system. We worked with ICE Officers and the DOJ to connect directly to the DOJ Court Scheduling system from the ICE User Interface. This allows the ICE Officer to save 7-13 minutes per court scheduling event. This system was developed in conjunction with DOJ and ICE personnel so that we had the most streamlined interaction possible, all while cutting down duplicate data entry to zero, while increasing data quality significantly.

All these systems were developed during the height of the COVID pandemic, with the Dev team traveling to the SW border every week for months, without a single person coming down with COVID. Being with the officers and agents in the field proved to be invaluable as the team could see and experience exactly what was happening on the ground, thus allowing them to work with end users to brain storm ideas on the best way to improve the processes and decrease the processing backlog.

Dev Technology AI Model Rejects 91% of Unacceptable EULA Clauses

CLICK HERE
image
Dev Technology AI Model Rejects 91% of Unacceptable EULA Clauses

Introduction

Dev Technology entered and won the GSA’s AI/ML Technical Challenge for automating the rejection of Enterprise User License Agreement (EULA) clauses which are unacceptable to US Federal contractual standards.  We competed against some 60 commercial, academic and individual entrants. Our solution used Deep NLP to achieve a 91% accuracy based on only 8000 example clauses. Our solution also explained its determination of acceptability by showing users examples of established acceptable and unacceptable clauses. Our solution was packaged as a machine-learning enabled API and a single-page web application, and can process EULA texts in commonly used formats.

Regulatory and legal acceptability of EULA Clauses is a ‘zero-tolerance’ style determination – no matter how much acceptable text may exist within a clause, if one part of it is identifiably unacceptable, the whole clause is considered unacceptable.  This differs from traditional sentiment analysis or topical classification.  Dev Technology’s solution breaks clauses into text ‘windows’ and decides if a window contains unacceptable language which can poison the whole clause.

In training our classification model, our team employed transfer learning from the Bidirectional Encoder Representations from Transformers (BERT).  BERT is an open source library which uses deep learning to encode linguistic knowledge about tens of thousands of words from millions of documents.  These encodings capture multiple contextual meanings and provide an excellent transfer learning basis to address NLP tasks which may otherwise suffer from small training set sizes.

Automation of Clause Rejection

Dev Technology’s AI model reduced the time EULA clause analysis takes from hours to seconds.  The two primary metrics for text classification are precision – how often are the clauses the model identifies as unacceptable really so; and recall – how often does the model catch the unacceptable clauses which are present in a EULA.  The higher the precision of a model, the less often a human analyst must reverse an automated decision.  The higher the recall of a model, the more confident a human can be that all the unacceptable clauses were found.  Dev Technology’s model scored over 90% in both metrics, for a combined 91% overall accuracy score.

Explanatory AI

High-quality AI classifiers not only achieve a good accuracy rate, but can also explain their decisions using human-understandable data.  Dev Technology’s EULA clause classifier provides evidence of known acceptable and unacceptable clauses from standard reference materials which most closely match the clause it made its decision about.  This similarity calculation is performed at classification time, and stored as metadata for use in a user-interface or batch dataset creation.

AI Model Delivery

The EULA clause classifier was developed using traditional data science tools such as open source Python machine learning libraries, the large BERT NLP model and Jupyter Notebooks development environment.  All of these tools greatly accelerate the AI/ML development process, but none of them are acceptable deliverables to an operational environment supporting business users.  Dev Technology packaged the final, tuned model as a RESTful API, with an accompanying single-page web application.  The clause classifier can be invoked either one document at a time, or as a batch process over many documents.  The results can be visualized using HTML and JavaScript, or can be saved as a dataset compatible with all major business intelligence tools.

Overall Results

Dev Technology’s surprise win in their first entry in a Federally sponsored AI/ML technical challenge validates both their domain knowledge in AI/ML and their ability to rapidly prototype, refine and deliver advanced technology applications that support their customers’ business needs.  Dev Technology meets the mission-critical needs of government agencies by exceeding our clients’ expectations through partnership, a commitment to teamwork, collaboration, and valuing our employees.

The Dev Technology Team develops several systems to address the backlog of processing for the Southwest Border Initiative (SWBI).

CLICK HERE
image
The Dev Technology Team develops several systems to address the backlog of processing for the Southwest Border Initiative (SWBI).

The Dev Technology Team, as part of our ICE ERO Consolidated contract worked with the ICE ERO OCIO to develop several systems to address the backlog of processing across the US Southern Border, together these systems fall under the Southwest Border Initiative (SWBI). Over the past year the team traveled to the Southwest border to sit on site, under bridges and in processing centers with ICE Officers and CBP Agents so we could see how systems were being used in the field, and to get real world feedback and suggestions from end users in high stress situations on improvements they needed in order to allow them to do their jobs faster, more accurately, and ultimately allow them to reduce the backlog of processing for non-citizens.

For the Case Acceptance System (CAS) we developed a system used by both CBP and ICE that allows CBP to request a Case Review by ICE before a non-citizen is moved from CBP Custody. The process involved a CBP Agent scanning over 40 pages of documents, attaching the newly scanned file to an email, and emailing the file to ICE Officers. ICE would then review the case files and send back information via email when the files needed to be changed and updated, before a decision could be made on a case. This process took on average 11 hours to complete. Our goal was to get that time down to under 2 hours. The team worked side by side with Agents and Officers in the field to develop a system from inception to Production in 5 weeks. The CAS Application reduced the time it took for case referrals, reviews, and decisions down to, on average, 52 minutes. With a vast majority of decisions coming in under 30 minutes. The CAS Application has now processed over 120K cases saving the end users hundreds of thousands of man hours. This allows the Agents and Officers to focus on other responsibilities.

The ATD Enrollment application was developed in very much the same fashion, the Dev team sitting with ICE Officers in soft sided facilities at the border to view firsthand the business processes and gather feedback on improvements from the end users. The ATD Enrollment application took an existing application, reengineered it with simplified workflow and connected it to an external contractor application. This application was developed from inception to production in 4.5 weeks. This gave the ICE Officers a 68%-time savings per enrollee, along with saving the contractor 50% of the time it normally took to enroll a noncitizen into the Alternate to Detention systems. The system has also cut data re-entry to zero and has drastically improved data quality across all systems. The time savings seen by the ATD Enrollment application in just the first 6 months of its deployment, at just one Southwest border site, saw a time savings of over 24,000-man hours or processing time.

The EOIR Court Scheduler Application looked to streamline the process of interacting with the Immigration Court Scheduling system. We worked with ICE Officers and the DOJ to connect directly to the DOJ Court Scheduling system from the ICE User Interface. This allows the ICE Officer to save 7-13 minutes per court scheduling event. This system was developed in conjunction with DOJ and ICE personnel so that we had the most streamlined interaction possible, all while cutting down duplicate data entry to zero, while increasing data quality significantly.

All these systems were developed during the height of the COVID pandemic, with the Dev team traveling to the SW border every week for months, without a single person coming down with COVID. Being with the officers and agents in the field proved to be invaluable as the team could see and experience exactly what was happening on the ground, thus allowing them to work with end users to brain storm ideas on the best way to improve the processes and decrease the processing backlog.

|
Our Clients Include

It’s fun coming to work every day to solve challenging problems with colleagues who you respect and consider friends.

Paige Jill
Senior Business Development Manager

Insights & News