AI tools are increasingly being developed and used in the legal sector generally, to assist in a wide range of tasks. These include drafting correspondence, conducting document reviews, and managing and amending legal documents (e.g., in redline format). The intended benefit of these tools is predominantly increased efficiency. However, without regular and rigorous evaluation, such AI tools could potentially generate flawed or inaccurate outputs, which would negate this efficiency advantage. Conversely, by subjecting AI systems to comprehensive testing, legal professionals can have greater confidence in the outputs generated.
In the legal sector generally, AI evals may serve to:
- Identify and address biases that may be present in a specific AI model's training data or algorithms.
- Ensure compliance with evolving legal standards and regulations across different jurisdictions.
- Verify a specific AI model's ability to handle complex legal language and concepts accurately.
- Test the tool's performance across various areas of law and types of legal documents.
When it comes to Intellectual Property (IP), AI tools may be subject to evals for the above purposes, with the addition of some IP-specific requirements. At Solve Intelligence, we recognize that robust evals are the foundation of delivering a dependable AI-driven platform that IP practitioners can rely on. But how might evals for IP-specific AI tools differ from those designed for general purpose legal AI tools? We explore this below.
Evals for Patent Drafting and Prosecution
The patent process involves several related but largely separate workflows or tasks that a patent attorney usually has to perform. Two of these workflows that are key to obtaining a useful patent are patent drafting and patent prosecution.
Patent Drafting: Ensuring Precision and Compliance
Patent drafting is a complex process that usually involves transforming an invention disclosure from an inventor into a detailed patent specification, including a description of the invention and a set of claims that define the legal scope of protection afforded to the patent, if and when it is granted. Although some sections may appear simple or short, drafting each section involves a multifaceted approach. Taking the claims as an example, patent attorneys usually have to consider:
- Novelty and inventive Step: Claims must be assessed against prior art to ensure they present a new and non-obvious invention.
- Scope optimization: The claim's breadth must be carefully set to provide comprehensive protection for the invention while remaining specific enough to be patentable.
- Jurisdictional compliance: The claim language must adhere to the specific requirements of the jurisdictions one intends to enter.
- Technical accuracy: Claims must precisely and accurately define the technical features that combine to form the invention.
- Clarity and conciseness: The language used must be unambiguous and clear.
- Anticipation of potential infringement: One may wish to draft claims in a manner that allows potential infringers to be easily identified.
- Strategic positioning: Claims should be formulated to align with the applicant's business objectives and overall IP strategy.
Given that an independent claim may be as short as 5 or 6 lines of text long, there’s a lot of points to consider when evaluating even a very limited AI output. Indeed, AI tools designed to assist in drafting patent specifications require continuous evaluation to ensure their outputs consistently meet these complex requirements and necessary legal standards. These tools must demonstrate proficiency in understanding technical subject matter, following established practice frameworks, all whilst adhering to legal requirements (including updates to the law).
With respect to drafting, evals can be used to test:
- Accuracy in Technical Description: AI tools need to accurately convert complex technical concepts into clear, legally precise language. Evals can help ensure this process is accurate and avoids hallucinations.
- Adherence to Formal Structures: AI tools need to be capable of producing content that adheres to specific structural requirements of patent documents. Evals can test the AI's ability to consistently organise sections like the background, summary, detailed description, and claims properly.
- Legal Compliance: it would be beneficial for AI tools to stay up-to-date with the latest changes in law. Evals can help verify that the AI remains compliant with evolving patent laws and regulations (e.g., changes to the list of 101 examples, or newly published guidance).
- Consistency Across Technical Fields: AI tools should be able to maintain accuracy and quality across various technical fields. Evals can be designed to test the AI's performance across a wide range of subject matters.
- Error Reduction: AI tools may initially have potential weaknesses or biases in their output. Evals can help identify these issues, allowing developers to refine the system and minimise errors that could compromise the quality of drafted patent specifications.
- Continuous Improvement: AI systems benefit from regular refinement and improvement. Evals can create a feedback loop that allows for the addition and testing of new algorithms or models, ensuring the overall platform becomes increasingly reliable and sophisticated over time.
Patent Prosecution: Enhancing AI's Legal Arguments
Patent prosecution is the process of assessing a patent application to check that it meets all necessary requirements to be granted. This process usually takes the form of a negotiation (or argument) with a patent office, usually between a patent attorney and a patent office Examiner.
AI tools such as Solve's Patent Prosecution Copilot can assist patent attorneys by analysing office actions, generating response strategies, and drafting complete responses. However, due to the legal complexity involved, and the negotiation-based nature of the process, it is important that AI tools are regularly evaluated for their ability to deliver persuasive arguments that are legally sound.
Evals for AI tools in prosecution may focus on testing several key areas:
- Understanding office actions: AI tools may be able to accurately parse office actions for the objections, actions and associated deadlines they include. Evals can help ensure this process is accurate, even for implicit deadlines.
- Generating response strategies: AI tools may be capable of analysing the examiner's objections and formulating effective response strategies according to the relevant law. Evals can help to iteratively improve and build on these response strategies over time.
- Drafting response sections: AI tools may be capable of drafting sections or all of an office action response. Evals can help to ensure that these sections are drafted coherently, and that any citations (for example, to the office action, prior art, or case law) are accurate and strengthen the arguments being made.
The Benefits of Evals for Patent Drafting and Prosecution
Having reviewed possible use cases of evals in testing and improving on AI tools for drafting and prosecuting patents, what benefits might you expect to see by implementing these?
- Improved Accuracy: By implementing evals, potential issues in AI outputs can be identified and corrected in the underlying algorithms or models, ensuring that risk of error is minimised.
- Enhanced Domain Customization: Evals allow for assessment of AI performance across varied technical domains. These metrics can be used to iterate on the underlying algorithms to ensure that AI tools can effectively handle the specific challenges of different technical fields.
- Better Jurisdictional Adaptation: Through evals, AI tools can be fine-tuned to adapt to the requirements of different patent offices worldwide. This is particularly beneficial given the significant differences in patent laws between certain jurisdictions.
- Increased Trust: Continuous evaluations provide transparency into AI performance and improvement, helping to build trust among patent attorneys. By sharing eval results, AI developers can demonstrate the reliability and tailored nature of their tools, increasing confidence in their use among patent professionals.
- Faster & More Efficient Development: Evals provide valuable visibility into the effectiveness of new features, allowing developers to quickly determine whether these additions are working as intended. This feedback loop enables more rapid iteration and refinement of AI tools, leading to faster development cycles and more efficient allocation of resources towards features that demonstrably improve performance.
Evals at Solve Intelligence
At Solve Intelligence, we place a strong emphasis on running thorough evaluations for our AI tools, especially when testing and introducing new features. Our Patent Copilots are designed to assist patent professionals draft and prosecute patent applications efficiently. At each stage, our Copilots work with the patent professional, keeping them in the driving seat, giving them the ability to reap AI's benefits while mitigating its associated challenges. For drafting and prosecution, our platform includes:
- Patent Drafting Copilot: This tool helps patent attorneys draft patent specifications based on invention disclosures, by leveraging AI in several different ways. Continuous evals help refine the tool's ability and allow us to add new features quickly.
- Patent Prosecution Copilot: Designed to support attorneys during the prosecution phase, this tool helps analyse office actions, generate response strategies, and write office action responses. By using rigorous evals, we are able to improve on the quality of argumentation and amendment, helping attorneys respond more effectively and efficiently to office actions.
.png)
