Custom Entities (Regex)

Use regex custom entities when the values you need to detect follow a stable structure, such as case references, medical record numbers, invoice IDs, employee IDs, or organization-specific account codes.

Regex detection matches the patterns you define. It is best for structured values with predictable prefixes, separators, or lengths. For concepts that need semantic judgment, use description-based custom entities instead.

Simple `expressions` field

Use expressions when one entity type can share the same score, context words, and operator across all of its regex patterns:

SELECT app_public.mask(
  discharge_summary,
  OBJECT_CONSTRUCT(
    'entity_config', OBJECT_CONSTRUCT(
      'CASE_REF', OBJECT_CONSTRUCT(
        'expressions', ARRAY_CONSTRUCT('REF-\\d{4,6}'),
        'score', 0.9,
        'operator', 'constant',
        'operator_params', OBJECT_CONSTRUCT('value', '[CASE_REF]')
      ),
      'MRN', OBJECT_CONSTRUCT(
        'expressions', ARRAY_CONSTRUCT('MR-\\d{4}-\\d{5}'),
        'score', 0.95,
        'operator', 'hash'
      )
    )
  )
) FROM app_public.sample_records WHERE record_id = 2;

Field	Required	Description
`expressions`	Yes	Non-empty array of regex strings. Any match counts as a detection for this custom entity.
`score`	No	Base confidence `0.0`-`1.0` for all expressions. Default `0.75`.
`context`	No	Array of words that can boost confidence when they appear near a match.
`threshold`	No	Per-entity minimum confidence `0.0`-`1.0`. Overrides the global threshold for this entity.
`operator`	No	Operator to apply. Defaults to `type_numbered`.
`operator_params`	No	Per-operator params.

Context words boost precision

Add context when a regex is broad enough to collide with other IDs or built-in entity types. Context words near a regex match can raise the confidence score.

OBJECT_CONSTRUCT(
  'NPI_NUMBER', OBJECT_CONSTRUCT(
    'expressions', ARRAY_CONSTRUCT('\\d{10}'),
    'score', 0.8,
    'context', ARRAY_CONSTRUCT('NPI', 'npi', 'provider'),
    'operator', 'mask'
  )
)

Without context, a 10-digit string can be hard to distinguish from a phone number, invoice number, or other numeric identifier.

Advanced `patterns` field

Use patterns when one custom entity type has multiple formats that need different scores or context words. Each pattern object has its own regex, score, optional name, and optional context.

OBJECT_CONSTRUCT(
  'INVOICE', OBJECT_CONSTRUCT(
    'patterns', ARRAY_CONSTRUCT(
      OBJECT_CONSTRUCT(
        'regex', 'INV-\\d{4}-\\d{5}',
        'score', 0.95,
        'name', 'invoice_id',
        'context', ARRAY_CONSTRUCT('invoice', 'billing', 'INV')
      ),
      OBJECT_CONSTRUCT(
        'regex', 'BILL-\\d{6}',
        'score', 0.9,
        'name', 'bill_id',
        'context', ARRAY_CONSTRUCT('bill', 'billing')
      )
    ),
    'operator', 'constant',
    'operator_params', OBJECT_CONSTRUCT('value', '[INVOICE]')
  )
)

Pattern field	Required	Description
`regex`	Yes	Regex string for this pattern variant.
`score`	Yes	Base confidence `0.0`-`1.0` for this pattern variant.
`name`	No	Label used for this pattern variant in detection details.
`context`	No	Array of context words for this pattern variant.

Do not set top-level score or context when using patterns; set them inside each pattern object instead.

Validation rules

Agent Mask validates custom regex configuration before running detection:

Use either expressions or patterns for a custom entity, not both.
expressions and patterns arrays cannot be empty.
score and threshold values must be between 0.0 and 1.0.
Regex strings must be valid and no longer than 500 characters.
Regex syntax must be RE2-compatible. Features such as lookaround and backreferences are rejected.
DEFAULT can set fallback operator behavior, but it cannot define custom detection fields such as expressions or patterns.
Unknown fields are rejected so typos fail fast instead of silently changing detection behavior.

Tuning guidance

Prefer specific, bounded patterns such as \\bREF-\\d{4,6}\\b over broad patterns such as \\d+.
Include literal prefixes, separators, and length constraints when your IDs have them.
Use context for generic numeric or alphanumeric IDs that may appear in non-sensitive text.
Start with a high score for strict formats, then adjust threshold after reviewing false positives and misses in analyzer_results.
Use allow_list for known safe values that match a custom pattern but should remain visible.
Keep custom entity names descriptive and stable, such as CASE_REF, CLAIM_ID, or EMPLOYEE_ID, so downstream reviewers can understand ledger entries.

Limitations

Regex custom entities only detect formats you encode. They will not infer unseen prefixes, OCR mistakes, inconsistent spacing, transposed characters, or semantic categories that do not have a stable pattern. Broad expressions can also match ordinary text, so validate custom patterns on representative samples before using them in production workflows.

Combining with description-based

Mix regex, description-based, and built-in entities in one call:

SELECT app_public.mask(
  discharge_summary,
  OBJECT_CONSTRUCT(
    'entity_config', OBJECT_CONSTRUCT(
      'DEFAULT', OBJECT_CONSTRUCT('operator', 'type_numbered'),
      'PERSON', OBJECT_CONSTRUCT('operator', 'faker'),
      'MEDICATION', OBJECT_CONSTRUCT(
        'description', 'prescription drug names',
        'operator', 'constant',
        'operator_params', OBJECT_CONSTRUCT('value', '[RX]')
      ),
      'MRN', OBJECT_CONSTRUCT(
        'expressions', ARRAY_CONSTRUCT('MR-\\d{4}-\\d{5}'),
        'score', 0.95,
        'operator', 'hash'
      )
    )
  )
) FROM app_public.sample_records WHERE record_id = 2;