Custom content moderation: apply your own policy, learned from your moderators' decisions

• Your lines are specific. "Is this investment advice or general education?" is decided by your compliance team's past calls, not by a general definition.
• It's consistent. The same message gets the same decision every time, which matters when you have to explain a decision.
• It fits an API workflow. Call the published classifier from your application; measure latency and check allowances against your workload.
• You can see how good it is. The report shows how often it agreed with your moderators on messages it never saw, rule by rule.

It reads the text to check: a user's message before your AI answers, or your AI's answer before the user sees it.

It answers with your policy's outcomes, for example:

• Allowed
• Off-topic for this product
• Needs a disclaimer
• Not allowed under our policy: investment advice
• Send to a moderator

With a confidence. Set automation rules based on your risk tolerance and evaluation results. Confidence alone is not proof that content is safe; hold uncertain cases for a person.

1. Export moderated examples

One row per message or answer, with the decision your moderators or compliance team made. Include plenty of allowed examples, not just the blocked ones: the model has to learn where the line is, from both sides.

2. Train and read the report

Upload and train an AI Classifier Model. Look at the mistakes rule by rule. Confusion with “Allowed” may reflect missing examples, ambiguous rules or model errors. Check both the training decisions and the policy.

3. Run it before and after your AI

• Before: check the user's message; decline or redirect off-topic and disallowed ones.
• After: check your AI's answer; add a disclaimer or hold it for a person.

Call the model through the Prediction API. On plans with the Review Queue, unsure answers wait there for your moderators.

4. Retrain when the policy changes

Add the new decisions and retrain. The model only knows the policy it was taught.

Your application must implement the hold, block or redirect action. A prediction entering the Review Queue does not stop your application from displaying it. API access and the Review Queue require Essentials or Business.

Use examples you are permitted to upload under the Terms of Service, and remove restricted personal data from moderation logs before sending them.

• It isn't a general safety filter. Keep your provider's safety systems, and dedicated tools for violence, self-harm, sexual content and similar. This handles your own rules on top of them.
• It doesn't explain its decisions. It gives a category and a confidence; your moderators give reasons.
• It learns your past decisions, inconsistencies included. Review the training examples before relying on it.

Use reviewed examples that reflect your current rules. Check errors by category and sample confident answers as well as uncertain ones. Confidence is a model score, not a guarantee. Keep mandatory safety and access rules in your application. Best practices for training and review →.