By: Rachel Fernandez, Director of Sales Engineering
Many clients are asking questions about integrating generative AI technologies into their existing workflows. Existing tools like Relativity’s aiR for Review promise expedited review and classification of documents, with a low entry commitment. Most of the marketing emphasizes the ease of prompt drafting and its simple set up. Below are some observations of Relativity’s aiR for Review and some general thoughts of the current version of this tool.
What it Does Well
Set Up Ease
The tool provides prompt instruction and examples for each set up step. Since the tool uses a large language model (LLM), it prefers simple, natural language prompting, making setup easy to follow.
Dashboard
The dashboard has integrated filtering and toggling between criteria and classification choices. You can focus on subpopulations like your most relevant documents, filter them through the aiR scores and quickly read through rationales and citations from the dashboard.
Considerations
Substantive Lift
Relativity markets the tool as a user-friendly entry into generative AI for review workflows. Yet, the training and substantive knowledge required to get decent results is considerable. The more knowledge you give the model to learn about your case in the prompting phase, the better results you’ll receive. The inverse is, if you don’t have many details about your case like it’s important dates and actors, the insights from this tool may be expensive and rudimentary. This is not an out-of-the-box solution that will give you data insights as a starting point to your discovery.
Prompt engineering is still considered best practice for the tool, even though the marketing will have you believe otherwise. Using formatting techniques like bulleted lists, strategic capitalization, avoiding jargon, explaining acronyms, and simplistic language focused on positive phrasing will greatly impact your results for the better. The reality is that prompt engineering will reduce your overall cost by reducing the number of iterations you need to run.
Data Limitations (Types & Size)
Generative AI tools in eDiscovery still rely on traditional analytics to run. This means that the same limitations we encounter with continuous active learning (CAL) and other technology assisted review (TAR) tools are still relevant to aiR for Review. Documents with poorly extracted text like images, spreadsheets, and database files are likely not going to return good insights or accurate classifications.
Another major limitation of this tool is the 150 kB limit. Documents over 150kB will likely error and not be analyzed by aiR for Review. This is much smaller than the 30MB exclusion for indexing when using other conceptual or structural analytics tools in Relativity. This small limitation can affect insights into data and will require manual review.
Output Limitations
The tool does not have any preview capabilities. To see the insights from changes in your prompt, you must run your prompt over your chosen population, incurring per document fees at every analysis.
Currently, all insights live within the aiR for Review tool and dashboard. To leverage the insights from the tool in your saved searches, you need to manually propagate the coding and scores to the documents in your universe. The only option the tool gives you is to save the documents as lists. You can, however, filter down to specific categories like Relevant or Borderline and save those sub populations as lists. But if you want to leverage the results in your saved searches, we have found it easiest to port over the coding into fields you can leverage outside of the tool.
Manual Validation
Unlike Relativity’s CAL tools or Review Center, there are no built-in validation tools for aiR for Review at this time. To defensively validate your results, you will need to pull statistical samples and calculate your precision and recall manually.
Summary
Relativity’s aiR for Review is a great tool and introduction to generative AI functionality. The tool’s main purpose is to classify documents quickly, based on the prompts from the user. This tool is best used on compatible data when the user already has a strong foundational knowledge of the substance of the matter and the types of data they are looking for.
This tool is not great at providing initial insights into the data as it heavily relies on details in the required prompts. The less you know about the data, the less detail you can provide in the prompts leading to more prompt iterations and overall cost. Take time to consider the costs and benefits of generative AI tools in your current workflows. Partnering with a provider like Sandline can help cut through costs by leveraging our expertise to get you meaningful results faster.
