Introducing agentic AI workflows for document extraction

In today's data-driven business environment, organizations face the challenge of extracting valuable information from countless documents in different formats. Whether you're processing invoices, analyzing contracts, or reviewing financial statements, getting structured data out of these documents efficiently can be a major automation hurdle.

‍

We're excited to announce a significant enhancement to Sensible's document extraction capabilities with the introduction of agentic LLM workflows. At its core, document extraction is about transforming unstructured documents into structured, machine-readable JSON data that can power applications and business processes. While Sensible has always excelled at extracting structured data from documents, our new agentic workflows take this capability to the next level by enabling multi-step, intelligent processing pipelines that can adapt to document variations and derive deeper insights from extracted information.

‍

Multi-step LLM document extraction

Document automation often requires multiple data-extraction steps, especially when handling varied document layouts or complex data relationships. To address this need, we've added two powerful capabilities to our extraction toolkit:
‍

• The ability to chain LLM prompts together using the new source_ids parameter
• Conditional execution of prompts based on extraction results using the conditional method
  ‍

These features enable far more sophisticated document processing workflows than were previously possible. Rather than trying to extract all the data in a single LLM prompting step, you can now break complex extractions into logical sequences, with each step building on the results of previous extractions.

For example, say that you want to extract deposits and withdrawals from bank statements. Let’s say Sensible Bank changed their statement formatting in 2023, and you want to handle both formats uniformly:

‍

To handle these differences, you can use conditional field execution to chain prompts together depending on which format you’re handling:

We’ll look at implementing this example in JSON code later, but for now, let’s paint a broader picture of example use cases for agentic LLM data extraction from documents.

‍

Powerful use cases
‍

Data enrichment

One of the most powerful applications of chained LLM prompts is data enrichment. Consider extracting a list of business names and descriptions from a document, then using a follow-on prompt to classify each business as retail or non-retail. This two-step approach delivers more accurate results by focusing the LLM on specific data rather than asking it to search through an entire document.

Geographic analysis offers another compelling use case. You could extract addresses from a document, then use a chained LLM prompt to derive all unique cities present in those addresses. For example, use this approach when analyzing real estate portfolios or customer distribution reports.
‍

Schema normalization

When working with documents that contain similar information but vary in structure, conditional execution and chained prompts provide a powerful solution for normalization.

For example, insurance carriers update their forms over time while maintaining the same policy structure. With conditional execution, you can detect which form version is being processed and then apply specific extraction strategies tailored to each document structure. With chained prompts, you can then standardize data regardless of the source document's format. We’ll dive deeper into schema normalization in a JSON code example later in this post.

‍

Semantic normalization

Beyond structural differences, documents often use varying terminology to describe the same concepts. You can use chained LLM prompts to standardize these semantic variations.

For example, medical documents frequently contain diagnoses expressed in different terms. You can extract these diagnoses using terminology from the document, then use a chained LLM prompt to standardize them to ICD-10 codes. This ensures consistency across all processed documents, regardless of the original terminology.

In legal contexts, contract clauses may use different wording but serve the same purpose. By extracting contract clauses and then using a follow-on LLM prompt to categorize them into standard clause types, you can build more effective contract analysis tools that work across diverse document sources.
‍

How to use agentic workflows

Let's explore a practical example of these new features in action.
‍

Using source_ids to chain LLM prompts

Imagine you need to extract transaction data from a bank statement and then analyze it further. In this example, we'll use a credit card statement that contains transaction details. We want to identify patterns, like the most frequent merchant and largest transaction.

Let's look at a sample credit statement:

This statement contains an "Account Activity" section with transaction details including dates, merchants, descriptions, and amounts. Using chained LLM prompts, we can first extract the entire transaction table and then run analytics on just that extracted data, rather than searching through the entire document.

Here's how you would implement this:

‍

{
  "fields": [
    {
      // First field: Extract the complete transaction table
      "id": "_transactions",  // Underscore prefix indicates this is an intermediate result
      "method": {
        // Use the nlpTable method to extract a structured table from the document
        "id": "nlpTable",
        // Tell the LLM what to look for in the document
        "description": "transaction history table",
        // Define the structure of the table we want to extract
        "columns": [
          {
            // First column: transaction dates
            "id": "transaction_date",
            "description": "date of transaction"
          },
          {
            // Second column: merchant information
            "id": "merchant_name",
            "description": "merchant"
          },
          {
            // Third column: transaction descriptions or categories
            "id": "transaction_description",
            "description": "description or category of transaction"
          },
          {
            // Fourth column: transaction amounts (required field)
            "id": "transaction_amount",
            "description": "amount",
            // Specify that this should be treated as currency with accounting conventions
            "type": "accountingCurrency",
            // Mark this as required - entries without amounts will be excluded
            "isRequired": true
          }
        ]
      }
    },
    {
      // Second field: Analyze the extracted transaction data using a chained prompt
      "method": {
        // Use queryGroup to ask multiple questions about the data
        "id": "queryGroup",
        // The key new feature: source_ids chains prompts by pointing to previously extracted data
        // instead of searching the whole document
        "source_ids": [
          "_transactions"  // Reference the transactions table we extracted above
        ],
        // Disable confidence signals for cleaner output
        "confidenceSignals": false,
        // Define the specific questions we want to ask about the transaction data
        "queries": [
          {
            // Query to find the most frequently appearing merchant
            "id": "freq_merchant_chained_query",
            "type": "string",
            "description": "What is the most frequent merchant?"
          },
          {
            // Query to find the largest transaction amount
            "id": "max_transaction_amount_chained_query",
            "description": "what is the maximum transaction amount?"
          }
        ]
      }
    }
  ]
}

‍
In this example, we first extract the transaction table from the document using the NLP Table method. Then, we use the Query Group method with source_ids pointing to our extracted transactions. The LLM only analyzes the extracted transaction data, not the entire document, to find the most frequent merchant and maximum transaction amount.

This approach provides more accurate results because it focuses the LLM on the specific data we want to analyze rather than requiring it to search through the entire document and potentially find irrelevant information.
‍

Using the Conditional method for document structure variations

The Conditional method allows you to handle document structure variations within the same document subtype, or “config”.  For Sensible users, this represents an increase in code-writing efficiency. Instead of authoring a new “config” for small variations, you can handle them in a single config. It also opens up powerful conditional logic for chaining prompts.

For example, let’s say the Sensible Bank changed its statement format, with some versions presenting transactions in separate tables while newer versions combine them into a single table:
‍

The preceding image shows two different formats of the same bank's statements. The 2023 format (left) uses a single combined "Transaction History" table that includes both deposits and withdrawals. The pre-2023 format (right) separates transactions into distinct "Deposits" and "Withdrawals" tables.

Using conditional execution, we can detect which format we're dealing with and apply the appropriate extraction strategy. Here’s the flow for the code we’ll be implementing:

‍

And here’s the actual code:
‍

{
  "fields": [
    {
      // First field: Determine which format version we're dealing with
      "method": {
        // Use queryGroup to ask a single question about the document structure
        "id": "queryGroup",
        "queries": [
          {
            // This query will return "yes" or "no" to identify the document format
            "id": "format_version",
            "description": "Is this the 2023 revised statement format that combines transactions into one table?"
          }
        ]
      }
    },
    {
      // Second field: Apply conditional logic based on the document format
      "method": {
        // Use the conditional method to branch the extraction logic
        "id": "conditional",
        // Define the condition using JsonLogic - checks if format_version equals "yes"
        "condition": {"==": [{"var": "format_version.value"}, "yes"]},
        // Fields to extract if the condition passes (2023 format with combined table)
        "fieldsOnPass": [
          {
            // Extract the single combined transactions table
            "id": "transactions",
            "method": {
              "id": "nlpTable",
              // Tailor the prompt for the combined table format
              "description": "Combined transaction history table with both deposits and withdrawals"
              // Additional configuration specific to the combined table format would go here
            }
          }
        ],
        // Fields to extract if the condition fails (pre-2023 format with separate tables)
        "fieldsOnFail": [
          {
            // Extract from separate deposits table
            "id": "deposits",
            "method": {
              "id": "nlpTable",
              // Tailor the prompt for the separate tables format
              "description": "Deposits table"
              // Additional configuration specific to the separate table format would go here
            }
          },
{
            // Extract from separate withdrawals table
            "id": "withdrawals",
            "method": {
              "id": "nlpTable",
              // Tailor the prompt for the separate tables format
              "description": "Withdrawals table"
              // Additional configuration specific to the separate table format would go here
            }
          }

        ]
      }
    },
    // You could add a third field here that uses source_ids to chain prompts and transform
    // the extracted transactions, regardless of which format they came from. For example, sort transactions by size instead of date.
    // This ensures consistent output structure despite different input formats
  ]
}

‍
This configuration first identifies the statement format, then uses JsonLogic to check if it's the 2023 revised version, and finally applies the appropriate extraction strategy based on the result. The document structure itself guides the extraction process, allowing for a more dynamic and adaptable approach.
‍

Putting It All Together

For the most complex documents, you can combine both techniques. For example, you might first use a query to detect document structure variation, then use conditional execution to choose the appropriate extraction strategy, extract structured data with methods tailored to the document structure, and finally use chained prompts to further analyze and transform the extracted data.

This workflow mirrors how a human might process a complex document: identify the specific format variant, apply the right approach to extract information, and then analyze that information to derive deeper insights.
‍

Conclusion

The addition of agentic LLM workflows represents a significant step forward in document extraction capabilities. By enabling chained LLM prompts and conditional execution, Sensible now provides developers with powerful tools to tackle even the most complex document automation challenges.

We're excited to see what you'll build with these capabilities. To get started with agentic workflows and learn how they can help automate your document processing needs, book a demo or check out our managed services for customized implementation support.

For more detailed information, check out our documentation on conditional execution and the source_ids parameter in our developer docs.

‍