OpenAI Privacy Filter Benchmark

Agent Mask beat OpenAI Privacy Filter in a 50,000-example PII detection benchmark.

On this English PII detection benchmark, Agent Mask reached 96.9% F1 versus 74.2% for OpenAI Privacy Filter. Privacy Filter is an open-weight model for context-aware PII detection in unstructured text; Agent Mask detects PII and returns redacted output, stable replacements, and review metadata in one private de-identification workflow.

Start Snowflake Trial Get Self-hosted Trial Key

Start a trial on Snowflake Marketplace, or get a self-hosted trial key for Docker.

OpenAI Privacy FilterOpen modelOpen-weight PII detection model from the same AI lab that brought us ChatGPT
Privacy Filter outputDetections onlyIt returns PII detections; your team still has to turn them into redacted output, stable replacements, review metadata, and writeback
Agent MaskRedacted outputFinds PII and returns de-identified text, replacements, and review details in Snowflake or Docker
Example masking call 
1SELECT agent_mask_en.app_public.mask(
2  [ticket_note, chat_log]
3) AS redacted
4FROM support_tickets;
Preview Same entity. Same replacement. Every column.
Input column Text
ticket_note Billing question from Maya Chen PERSON_1 about invoice INV-1042. Assigned to Alex Patel PERSON_2 . Contact: maya.chen@example.com EMAIL_ADDRESS_1 or 415-555-0198 PHONE_NUMBER_1
chat_log 10:42 Alex PERSON_2 : Maya PERSON_1 never got the invoice email.
10:43 Maya Chen PERSON_1 replied from maya.chen@example.com EMAIL_ADDRESS_1 and asked Alex Patel PERSON_2 to call 415-555-0198 PHONE_NUMBER_1
OpenAI's published score vs this benchmark

OpenAI's published score did not hold up.

OpenAI reports a strong Privacy Filter score in its release. This local benchmark tested how OpenAI Privacy Filter and Agent Mask handled 50,000 English PII examples under the same setup. press release

Privacy Filter result
74.2%

F1 on this PII detection benchmark.

What they promised
OpenAI published score
97.43% -23.2 pts difference

OpenAI reports this corrected PII-Masking-300k F1 score on its Privacy Filter release page. This test came in 23.2 points lower.

Agent Mask result 96.9%

F1 on this benchmark, with de-identified output and review details from one private workflow.

Benchmark speed

Agent Mask ran faster

11.7x

faster in this run, measured on a consumer RTX 3080. Production speed depends on deployment and batching.

50,000 English PII examples across names, locations, dates, emails, phone numbers, and URLs. Same test setup for both systems.

Benchmark proof

Agent Mask found the PII OpenAI missed.

The accuracy gap is not subtle. On this PII detection benchmark, Agent Mask outperforms OpenAI Privacy Filter on overall F1, PII found, and match quality. The aggregate numbers are the stronger signal: more PII found, fewer cleanup paths, and a response your team can use.

Accuracy at a glance

The scores point to the same operational outcome: Agent Mask misses fewer PII detections than OpenAI's Privacy Filter.

Overall F1 +22.7 pts
Agent Mask 96.9%
OpenAI 74.2%
PII found +28.7 pts
Agent Mask 99.9%
OpenAI 71.2%
Match quality +22.0 pts
Agent Mask 98.1%
OpenAI 76.1%

What comes back after detection

Privacy Filter gives you detections to wrap. Agent Mask returns de-identified text plus entity positions, a ledger, and request-time controls for custom entities and replacement behavior.

OpenAI Privacy Filter

A grouped detection result still needs application code for redaction, replacement consistency, review evidence, and writeback.

  • You host and call the model.
  • Your application turns detections into de-identified text.
  • Your team builds replacement consistency, review metadata, and writeback.
  • Custom entities and per-entity replacement rules need another layer.
[
  {
    "entity_group": "private_person",
    "score": 0.99,
    "word": "Maya Torres",
    "start": 0,
    "end": 11
  },
  {
    "entity_group": "account_number",
    "score": 0.86,
    "word": "CLM-1042",
    "start": 24,
    "end": 32
  }
]
{
  "columns": [
    {
      "deidentified_text": "[PERSON_1] filed claim 8f3a2c91 at ******************.",
      "analyzer_results": [
        {
          "entity_type": "PERSON",
          "start": 0,
          "end": 11
        },
        {
          "entity_type": "CLAIM_ID",
          "start": 24,
          "end": 32
        },
        {
          "entity_type": "LOCATION",
          "start": 36,
          "end": 54
        }
      ]
    }
  ],
  "entity_ledger": [
    {
      "entity_type": "PERSON",
      "original": "Maya Torres",
      "replacement": "[PERSON_1]"
    },
    {
      "entity_type": "CLAIM_ID",
      "original": "CLM-1042",
      "replacement": "8f3a2c91"
    },
    {
      "entity_type": "LOCATION",
      "original": "Northbridge Clinic",
      "replacement": "******************"
    }
  ]
}
Agent Mask

A private de-identification response includes the output text, the evidence, and the controls needed for production workflows.

  • Cleaned text is ready for downstream workflows.
  • Review details show where each sensitive value appeared.
  • The ledger ties original values to the replacement used.
  • Different entity types can use different replacement behavior, from readable placeholders to masks, hashes, synthetic values, encryption, labels, or date shifts.

Representative response shapes are shortened for readability. The OpenAI example reflects grouped model output; Agent Mask field names come from the public response format.

Agent Mask wins on the numbers and the workflow.

OpenAI Privacy Filter is a useful open-weight model for PII detection. In this benchmark, Agent Mask found more PII and already includes the de-identification workflow that your team needs.

OpenAI Privacy Filter fit

You want an open model to host and customize.

OpenAI Privacy Filter can make sense if your team wants an Apache-2.0 model and has the engineering time to host it, tune it, monitor it, and connect it to a redaction workflow.

Agent Mask fit

Agent Mask finds more PII, then gives you the workflow.

The benchmark shows Agent Mask found more PII. It also gives you what a redaction workflow needs: de-identified output, review evidence, consistent replacements, custom PII/PHI rules, and private deployment as a Snowflake Native App or self-hosted container.

PII detection is only the first step.

OpenAI Privacy Filter focuses on context-aware PII detection in unstructured text. Agent Mask detects PII, replaces it, keeps replacements consistent, returns review metadata, and runs in Snowflake or a self-hosted container.

Redacted output, not model plumbing

OpenAI Privacy Filter returns detections. Your team still has to turn them into masked text, writeback, and review evidence. Agent Mask returns de-identified text, replacement details, and review details in one workflow.

Add your own sensitive terms

Define custom PII, PHI, IDs, or policy-specific terms with plain-English descriptions or regular expressions. Privacy Filter uses a fixed taxonomy; custom entities require another detector or application rules.

Run where the data lives

Use Agent Mask in Snowflake or a self-hosted container so PII does not have to move into a separate model service.

OpenAI Privacy Filter vs Agent Mask.

Compare an open-weight PII detection model with a private de-identification workflow that includes redacted output, review metadata, replacement controls, custom entities, document/file support, and deployment choices.

Dimension
OpenAI Privacy Filter
Agent Mask
What it is
OpenAI Privacy FilterAn open Hugging Face model for finding PII.
Agent MaskA private redaction workflow for finding, replacing, and reviewing PII.
Where it runs
OpenAI Privacy FilterYour team hosts the model and connects it to the systems that hold PII.
Agent MaskRuns inside Snowflake or your own infrastructure, with no separate redaction service.
Benchmark performance
OpenAI Privacy Filter74.2% overall F1 and 71.2% of PII found on the 50,000-example benchmark.
Agent Mask96.9% overall F1 and 99.9% of PII found on the same benchmark.
What comes back
OpenAI Privacy FilterModel findings only. Your team turns those detections into masked text and audit metadata.
Agent MaskDe-identified text plus review details and a ledger of what changed.
Detection scope
OpenAI Privacy FilterLimited to the categories the model knows unless you add another detector.
Agent MaskBuilt-in PII plus custom terms defined in plain English or regex.
Replacement controls
OpenAI Privacy FilterNo built-in replacement step. Your application decides how each finding becomes masked text.
Agent MaskChoose placeholders, masks, fake values, hashes, encryption, and consistent aliases per request.

Reference material

This page uses OpenAI Privacy Filter release materials, local Agent Mask benchmark output, and Agent Mask docs as references.

OpenAI Privacy Filter model card

Hugging Face model card for OpenAI Privacy Filter, including intended use, Apache-2.0 license, English-focused details, and deployment cautions.

Read model card

OpenAI Privacy Filter release page reporting 97.43%

OpenAI release page where the 97.43% corrected PII-Masking-300k F1 score for Privacy Filter is reported.

Read OpenAI release page

OpenAI Privacy Filter PDF model card

OpenAI PDF model card with benchmark tables, intended-use framing, output shape, and limitations.

Read PDF model card

Agent Mask response format

Agent Mask docs for redacted text, review details, document details, and redaction details.

Read response docs

Agent Mask request parameters

Agent Mask docs for choosing what to find, adding custom PII, returning review details, and controlling replacements.

Read parameter docs

Agent Mask zero-egress architecture

Agent Mask documentation explaining how Snowflake Native App and self-hosted Docker processing stay inside your chosen runtime without sending payloads to Agent Mask.

Read zero-egress docs

Agent Mask self-hosted security

Agent Mask documentation for self-hosted data flow, network requirements, offline license verification, retention, and hardening.

Read self-hosted security docs

Related comparisons

Keep evaluating redaction options across the same private-runtime, data movement, pricing, and implementation questions.

Snowflake AI_REDACT alternative

Snowflake AI_REDACT alternative for long text, documents, and custom entities.

Compare Snowflake AI_REDACT with Agent Mask for PII redaction in Snowflake: long text, documents, custom entities, consistent replacements, and review metadata.

Read the comparison

John Snow Labs de-identification alternative

John Snow Labs handles clinical NLP. Agent Mask returns PHI de-identification with audit evidence.

Compare John Snow Labs with Agent Mask for Snowflake PHI de-identification, clinical notes, audit mapping, custom entities, and private deployment.

Read the comparison

Amazon Comprehend PII alternative

Amazon Comprehend PII redaction finds entities. Agent Mask turns them into usable, governable de-identified data.

Compare Agent Mask with Amazon Comprehend PII for teams that need private de-identification, custom entities, consistent replacements, and deployment in Snowflake or self-hosted infrastructure.

Read the comparison

OpenAI Privacy Filter alternative FAQ

Answers for teams comparing an open PII detection model with a private redaction workflow.

When is OpenAI Privacy Filter enough?

Privacy Filter can fit if you want an open model for English PII detection and your team is ready to host it, test it, and build the redaction workflow around it.

What do we still have to build around Privacy Filter?

Privacy Filter returns detections, not redacted output. To use it in production, your team still has to host and monitor the model, validate the results, write them back, and turn findings into masked text, stable replacements, and review evidence.

Can Privacy Filter handle custom PII or PHI terms?

Not as part of the model. It returns the categories the model knows; custom PII/PHI terms, internal IDs, and policy-specific strings require another detector or application rules.

Can Privacy Filter keep replacements consistent?

No. Privacy Filter identifies PII; it does not decide how each value should be replaced or keep aliases mapped across related text. Agent Mask can do that in the redaction request.

What did the benchmark show?

On the 50,000-example English PII benchmark, Agent Mask reached 96.9% F1 versus 74.2% for OpenAI Privacy Filter, found 99.9% of PII versus 71.2%, and reached 98.1% match quality versus 76.1%. The biggest category gap was location, but the headline result is the aggregate accuracy gap plus the production response shape.

When should we use Agent Mask instead of OpenAI Privacy Filter?

Use Agent Mask when you need production redaction, not just model detections. Agent Mask finds more PII in less time, while also returning de-identified output, replacements, and review details.

Finding PII isn't enough.

If your goal is a production de-identification workflow, you need private deployment, replacement controls, custom entities, redacted output, and review metadata, not just a standalone detection model.

Start Snowflake Trial Get Self-hosted Trial Key