Lab SSA -- Legal
The challenge
You are the Semantic Systems Architect for Caldwell Legal Partners, a mid-size law firm with 30 attorneys specializing in commercial contracts. The firm reviews approximately 60 contracts per month -- service agreements, licensing deals, NDAs, and vendor contracts. Currently, a junior associate performs a first-pass review of each contract, marking up risks and deviations from the firm's standard positions. This process takes 5-8 hours per contract.
The managing partner wants an AI-assisted system that produces a structured risk analysis for each incoming contract, highlighting clauses that deviate from the firm's standards, flagging potential legal risks, and identifying regulatory compliance gaps. The goal is to reduce the first-pass review to under 1 hour, so that associates can focus on substantive legal analysis rather than reading every line.
Your job is to design the complete semantic architecture: the clause taxonomy, the risk classification framework, the evidence protocol, and the evaluation suite.
What makes this hard
Legal analysis combines several challenges that are rare in other domains:
Precision of language matters enormously. In a customer support system, the difference between "we will try to fix it" and "we will fix it" is a matter of customer satisfaction. In a contract, the difference between "shall" and "may" is the difference between a binding obligation and a discretionary option. Your ontology must capture these linguistic distinctions and their legal significance.
Context determines meaning. A clause that says "the agreement may be terminated for convenience with 30 days notice" is perfectly reasonable in a month-to-month service agreement. The same clause in a 5-year exclusive licensing deal creates enormous risk because the licensor can walk away at any time. Your risk analysis must account for the context of the entire contract, not just the language of individual clauses.
The absence of a clause is as important as its presence. A contract that fails to include a limitation of liability clause, a force majeure clause, or a data protection clause may be more risky than a contract that includes unfavorable versions of these clauses. Your system must detect what is missing, not just analyze what is present.
Fabrication is unacceptable. If the system cites a case that does not exist, or cites a statute that was repealed, or references a regulation that applies to a different jurisdiction, the consequences can be severe. Unlike other domains where a hallucinated fact might cause mild confusion, a hallucinated legal citation can cause malpractice liability.
Deliverables
You must produce four artifacts. Each artifact has specific minimum requirements.
Deliverable 1: Clause taxonomy and risk classification
Design a comprehensive taxonomy for classifying contract clauses and a framework for assessing risk. Your taxonomy must include:
Clause types (minimum 12):
| Category | Clause types |
|---|---|
| Core commercial | Payment terms, pricing, scope of services, deliverables |
| Risk allocation | Liability limitation, indemnification, warranty, insurance |
| Governance | Governing law, dispute resolution, amendment, assignment |
| Duration and exit | Term, renewal, termination for cause, termination for convenience |
| Protection | Confidentiality, IP ownership, non-compete, data protection |
| Compliance | Regulatory compliance, anti-corruption, export control |
For each clause type, define:
- What the clause governs
- The firm's standard position (what the firm considers acceptable)
- Common risk patterns (what language or terms create risk)
- Severity when missing (is the absence of this clause a risk?)
Risk classification framework:
Design a risk model with at least three dimensions:
Risk assessment = f(legal_enforceability, financial_exposure, deviation_from_standard)
legal_enforceability:
- enforceable: clause is valid under governing law
- uncertain: enforceability depends on jurisdiction or interpretation
- unenforceable: clause conflicts with applicable law
financial_exposure:
- minimal: less than 10% of contract value
- moderate: 10-50% of contract value
- significant: 50-100% of contract value
- unlimited: no cap on exposure
deviation_from_standard:
- none: matches firm's standard position
- minor: slightly less favorable than standard
- significant: materially less favorable than standard
- major: opposite of firm's standard position
Each combination maps to an overall risk level (low, medium, high, critical) with a recommended action (accept, negotiate, reject, escalate to partner).
Deliverable 2: Evidence protocol
Every risk assessment and every recommendation must be supported by evidence. Design a protocol that defines what constitutes acceptable evidence and how it must be presented.
Evidence types:
- Contract language: direct quotes from the clause being analyzed, with section reference
- Legal authority: verified case citations or statute references from the governing jurisdiction
- Firm precedent: reference to the firm's clause library or prior deal positions
- Regulatory requirement: specific regulation text with applicability analysis
- Market standard: reference to industry benchmarks or survey data
Evidence requirements by risk level:
| Risk level | Minimum evidence required |
|---|---|
| Low | Contract language reference |
| Medium | Contract language + firm standard comparison |
| High | Contract language + firm standard + at least one legal authority or regulatory reference |
| Critical | Contract language + firm standard + legal authority + partner-level review required |
Verification rules:
- Every cited case must be verifiable against an authoritative legal database
- Every cited statute must include the version date to ensure it is current
- Every firm precedent reference must include a deal identifier that can be cross-checked
- If the system cannot find supporting authority, it must state "no supporting authority identified" rather than fabricate a citation
Deliverable 3: Risk classification and gap analysis
Design a systematic process for analyzing a contract and producing a risk map. The process must include:
Step 1: Structural decomposition
- Identify all clauses and classify them using your taxonomy
- Flag any expected clause types that are missing
- Identify cross-references between clauses (e.g., a termination clause that references the liability clause)
Step 2: Clause-level risk assessment
- For each clause, assess risk using your three-dimensional framework
- For each identified risk, provide the required evidence
- Flag language patterns that are commonly associated with risk (unlimited liability, unilateral modification rights, automatic renewal without notice)
Step 3: Contract-level risk assessment
- Aggregate clause-level risks into an overall contract risk profile
- Identify interactions between clauses that create compound risks (e.g., broad indemnification + no liability cap = effectively unlimited exposure)
- Identify missing protections and assess their impact on overall risk
Step 4: Recommendation generation
- For each high or critical risk, provide a specific recommendation (renegotiate clause language, add a missing clause, request a carve-out)
- For each recommendation, provide the evidence that supports it
- Organize recommendations by priority (must-negotiate vs. nice-to-have)
Deliverable 4: Evaluation suite
Create an evaluation suite with at least 30 synthetic contracts designed to test the system's accuracy and reliability.
Standard contracts (12 cases): Contracts with typical, market-standard terms. Expected outcome: few or no high-risk flags. Tests that the system doesn't over-flag standard language.
Example format:
Contract ID: STD-003
Type: SaaS agreement, 2-year term
Governing law: Delaware, USA
Parties: Enterprise buyer / SaaS vendor
Key terms: standard liability cap (2x annual fees), mutual indemnification,
30-day termination for convenience, GDPR-compliant DPA
Expected risk flags: 0 critical, 0-1 high, 2-3 medium
Expected missing clauses: none
Risky contracts (10 cases): Contracts with deliberately unfavorable terms, missing clauses, or ambiguous language. Expected outcome: the system should flag specific risks with appropriate severity.
Include scenarios such as:
- A contract with no liability cap
- A contract with unilateral modification rights
- A contract with an indemnification clause that is much broader than standard
- A contract missing a data protection addendum despite processing personal data
- A contract with a governing law that conflicts with where the parties operate
Jurisdictional variants (5 cases): The same contract type governed by different jurisdictions. Expected outcome: the system should identify jurisdiction-specific risks.
- A non-compete clause governed by California law (generally unenforceable) vs. Texas law (generally enforceable)
- A data processing agreement under GDPR vs. under CCPA vs. under a jurisdiction with no data protection law
Adversarial cases (3 cases): Contracts designed to test the system's resistance to hallucination and overconfidence.
- A contract in a niche area with limited precedent (the system should acknowledge limited authority, not fabricate it)
- A contract with contradictory clauses (the system should flag the contradiction, not resolve it silently)
- A contract in a language other than English (the system should flag the language limitation rather than provide a potentially inaccurate analysis)
Evaluation rubric
Your lab submission is evaluated across four dimensions. Each dimension is scored from 0 to 10.
Dimension 1: Conceptual precision (0-10)
| Score | Description |
|---|---|
| 0-2 | Clause taxonomy is vague or incorrect. Legal terms used improperly. Risk categories are undefined. |
| 3-4 | Basic clause types identified but taxonomy is incomplete. Risk framework exists but dimensions are poorly defined. |
| 5-6 | Taxonomy covers standard clause types with correct legal terminology. Risk framework has clear dimensions. Minor gaps in coverage. |
| 7-8 | Comprehensive taxonomy with accurate legal definitions. Risk framework captures multiple dimensions. Firm-specific standards well-integrated. |
| 9-10 | Taxonomy reflects how experienced commercial lawyers actually categorize contract clauses. Risk framework is nuanced, capturing enforceability, exposure, and deviation. Could be used by a real firm. |
Dimension 2: Explainability (0-10)
| Score | Description |
|---|---|
| 0-2 | Risk assessments provide no reasoning. No evidence cited. |
| 3-4 | Some reasoning provided but inconsistent. Evidence citations sporadic. |
| 5-6 | Every risk assessment includes a rationale and contract language reference. Some legal authority cited. |
| 7-8 | Every risk assessment includes a clear rationale, contract language, and appropriate evidence per the evidence protocol. A lawyer can understand and verify each assessment. |
| 9-10 | Risk assessments are attorney-ready. Evidence chains are complete and verifiable. Reasoning follows standard legal analysis structure (rule, application, conclusion). A partner could review the output and trust it as a foundation for client advice. |
Dimension 3: Hallucination control (0-10)
| Score | Description |
|---|---|
| 0-2 | No safeguards against fabricated citations. System may present AI-generated analysis as established legal fact. |
| 3-4 | Basic awareness of hallucination risk but no architectural controls. |
| 5-6 | Verification requirements defined for citations. Confidence levels included in outputs. |
| 7-8 | Hard constraints prevent unverified citations from appearing in outputs. System clearly distinguishes between verified authority and AI analysis. Adversarial test cases addressed. |
| 9-10 | Comprehensive anti-hallucination architecture. Every factual claim is traceable to a verifiable source. The system explicitly states when it cannot find supporting authority. Adversarial cases are handled safely (system acknowledges limitations rather than fabricating). |
Dimension 4: Utility for legal team (0-10)
| Score | Description |
|---|---|
| 0-2 | Output is not useful for practicing attorneys. Format is unclear. Recommendations are generic. |
| 3-4 | Output has some utility but requires significant rework. Risk priorities unclear. |
| 5-6 | Output provides a useful starting point. Risk priorities clear. Recommendations are actionable. |
| 7-8 | Output significantly reduces first-pass review time. Risk map is clear and comprehensive. Recommendations are specific and prioritized. Missing clause detection works. |
| 9-10 | Output is production-ready for a real law firm. An associate could use it as the foundation for a complete risk memo. Risk assessments align with how senior attorneys evaluate contracts. The system saves measurable time without sacrificing quality. |
Submission checklist
Before submitting, verify:
- Clause taxonomy includes at least 12 clause types with definitions and risk patterns
- Risk classification uses at least 3 assessment dimensions
- Evidence protocol defines requirements by risk level
- Evidence protocol includes verification rules that prevent fabricated citations
- Risk classification process covers both present clauses and missing clauses
- Compound risk detection is addressed (interactions between clauses)
- Evaluation suite contains at least 30 synthetic contracts
- Evaluation suite includes jurisdictional variants and adversarial cases
- All outputs include evidence chains and confidence levels
- System clearly states when it lacks supporting authority
What comes next
After completing this lab, you can use your deliverables as a foundation for your capstone if your organization operates in a legal context. The clause taxonomy, risk framework, and evidence protocol are directly applicable to real firm operations.
If you are pursuing other specializations, continue to SSA for B2B Support.