---
title: 內容審核
url: https://platform.claude.com/docs/zh-TW/about-claude/use-case-guides/content-moderation
description: 內容審核是在數位應用程式中維持安全、尊重且具生產力環境的關鍵環節。本指南將討論如何使用 Claude 來審核您數位應用程式中的內容。
---
> 請造訪[內容審核 cookbook](https://platform.claude.com/cookbook/misc-building-moderation-filter)，查看使用 Claude 實作內容審核的範例。
本指南著重於審核您應用程式中由使用者產生的內容。如果您正在尋找有關審核與 Claude 互動的指引，請參閱
[減輕越獄與提示注入](https://platform.claude.com/docs/zh-TW/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks)
。
## 使用 Claude 建構之前
### 決定是否使用 Claude 進行內容審核
以下是一些關鍵指標，說明您應該使用像 Claude 這樣的「large language model」（大型語言模型），即 LLM，而非傳統的機器學習（ML）或基於規則的方法來進行內容審核：
傳統的 ML 方法需要大量的工程資源、ML 專業知識以及基礎設施成本。人工審核系統的成本甚至更高。使用 Claude，您可以在明顯更短的時間內、以更低的成本，讓一套精密的審核系統上線運作。
傳統的 ML 方法，例如詞袋模型（bag-of-words）或簡單的模式比對，往往難以理解內容的語氣、意圖與上下文。雖然人工審核系統擅長理解語意，但需要時間來審查內容。Claude 結合了語意理解與快速做出審核決策的能力，同時滿足這兩項需求。
藉由其先進的推理能力，Claude 能夠一致地詮釋並套用複雜的審核準則。這種一致性有助於確保所有內容獲得公平對待，降低因不一致或帶有偏見的審核決策而損害使用者信任的風險。
一旦建立了傳統的 ML 方法，要加以變更是一項費力且需要大量資料的工作。另一方面，隨著您的產品或客戶需求演進，Claude 可以輕鬆適應審核政策的變更或新增，而無需大規模重新標註訓練資料。
如果您希望向使用者或監管機構提供審核決策背後的清楚說明，Claude 可以產生詳細且連貫的理由。這種透明度對於建立信任以及確保內容審核實務的問責性相當重要。
傳統的 ML 方法通常需要為每種支援的語言建立獨立的模型或進行大量的翻譯流程。人工審核則需要聘僱精通每種支援語言的人力。Claude 的多語言能力使其能夠對各種語言的工單進行分類，而無需獨立的模型或大量的翻譯流程，從而簡化面向全球客戶群的審核工作。
Claude 的多模態能力使其能夠分析並詮釋文字與圖像內容。這使它成為一項多功能工具，適用於需要同時評估不同媒體類型的環境中進行全面的內容審核。
所有 Claude 模型都經過訓練，具備內建的安全行為。這可能導致 Claude 無論使用何種提示，都會審核被視為特別危險的內容（符合
[可接受使用政策](https://www.anthropic.com/legal/aup)
）。例如，一個希望允許使用者發布露骨性內容的成人網站可能會發現，即使他們在提示中指定不要審核露骨性內容，Claude 仍會將露骨內容標記為需要審核。建議您在建構審核解決方案之前先行檢閱 AUP。
### 產生待審核內容的範例
在開發內容審核解決方案之前，請先建立應被標記的內容範例以及不應被標記的內容範例。請確保納入邊界案例以及內容審核系統可能難以有效處理的棘手情境。之後，檢視您的範例以建立一份定義明確的審核類別清單。 舉例來說，社群媒體平台所產生的範例可能包含以下內容：
```python Python
client = anthropic.Anthropic()
allowed_user_comments = [
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!",
]
disallowed_user_comments = [
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
]
# 用於測試內容審核的範例使用者留言
user_comments = allowed_user_comments + disallowed_user_comments
# 內容審核中被視為不安全的類別
unsafe_categories = [
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes",
]
```
```typescript TypeScript
const client = new Anthropic();
const allowedUserComments = [
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!"
];
const disallowedUserComments = [
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"
];
// 用於測試內容審核的範例使用者留言
const userComments = [...allowedUserComments, ...disallowedUserComments];
// 內容審核中視為不安全的類別
const unsafeCategories = [
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes"
];
```
```csharp C#
var client = new AnthropicClient();
string[] allowedUserComments =
[
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!",
];
string[] disallowedUserComments =
[
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
];
// 用於測試內容審核的範例使用者留言
string[] userComments = [.. allowedUserComments, .. disallowedUserComments];
// 內容審核中視為不安全的類別
string[] unsafeCategories =
[
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes",
];
```
```go Go
var client = anthropic.NewClient()
var allowedUserComments = []string{
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!",
}
var disallowedUserComments = []string{
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
}
// 用於測試內容審核的範例使用者留言
var userComments = slices.Concat(allowedUserComments, disallowedUserComments)
// 內容審核中視為不安全的類別
var unsafeCategories = []string{
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes",
}
```
```java Java
final AnthropicClient client = AnthropicOkHttpClient.fromEnv();
final List allowedUserComments = List.of(
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!");
final List disallowedUserComments = List.of(
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!");
// 用於測試內容審核的範例使用者留言
final List userComments =
Stream.concat(allowedUserComments.stream(), disallowedUserComments.stream()).toList();
// 內容審核中被視為不安全的類別
final List unsafeCategories = List.of(
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes");
```
```php PHP
$client = new Client();
$allowedUserComments = [
'This movie was great, I really enjoyed it. The main actor really killed it!',
'I hate Mondays.',
'It is a great time to invest in gold!',
];
$disallowedUserComments = [
'Delete this post now or you better hide. I am coming after you and your family.',
'Stay away from the 5G cellphones!! They are using 5G to control you.',
'Congratulations! You have won a $1,000 gift card. Click here to claim your prize!',
];
// 用於測試內容審核的範例使用者留言
$userComments = [...$allowedUserComments, ...$disallowedUserComments];
// 內容審核中被視為不安全的類別
$unsafeCategories = [
'Child Exploitation',
'Conspiracy Theories',
'Hate',
'Indiscriminate Weapons',
'Intellectual Property',
'Non-Violent Crimes',
'Privacy',
'Self-Harm',
'Sex Crimes',
'Sexual Content',
'Specialized Advice',
'Violent Crimes',
];
```
```ruby Ruby
CLIENT = Anthropic::Client.new
ALLOWED_USER_COMMENTS = [
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!"
]
DISALLOWED_USER_COMMENTS = [
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"
]
# 用於測試內容審核的範例使用者留言
USER_COMMENTS = ALLOWED_USER_COMMENTS + DISALLOWED_USER_COMMENTS
# 內容審核中視為不安全的類別
UNSAFE_CATEGORIES = [
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes"
]
```
要有效審核這些範例，需要對語言有細緻的理解。在留言 `This movie was great, I really enjoyed it. The main actor really killed it!` 中，內容審核系統需要辨識出「killed it」是一種比喻，而非實際暴力的跡象。相反地，儘管沒有明確提及暴力，留言 `Delete this post now or you better hide. I am coming after you and your family.` 仍應被內容審核系統標記。
不安全類別可以依據您的特定需求進行客製化。例如，如果您想防止未成年人在您的網站上建立內容，您可以將「Underage Posting」（未成年發文）加入類別中。
***
## 如何使用 Claude 審核內容
### 選擇合適的 Claude 模型
選擇模型時，考量資料量的大小非常重要。如果成本是考量因素，像 Claude Haiku 5.5 這樣的較小模型因其成本效益而是絕佳選擇。以下是針對每月接收十億則貼文的社群媒體平台進行文字審核的成本估算：
* **內容規模**
* 每月貼文數：1B
* 每則貼文字元數：100
* 總字元數：100B
* **估計 token 數**
* 輸入 token：28.6B（假設每 3.5 個字元為 1 個 token）
* 被標記訊息的百分比：3%
* 每則被標記訊息的輸出 token：50
* 總輸出 token：1.5B
* **Claude Haiku 5.5 估計成本**
* 輸入 token 成本：28,600 MTok \* $0.10/MTok = $2,860 USD
* 輸出 token 成本：1,500 MTok \* $0.50/MTok = $750 USD
* 每月成本：$2,860 + $750 = $3,610 USD
* **Claude Opus 5 估計成本**
* 輸入 token 成本：28,600 MTok \* $5.00/MTok = $143,000 USD
* 輸出 token 成本：1,500 MTok \* $25.00/MTok = $37,500 USD
* 每月成本：$143,000 + $37,500 = $180,500 USD
* **Claude Opus 4.8 估計成本**
* 輸入 token 成本：28,600 MTok \* $5.00/MTok = $143,000 USD
* 輸出 token 成本：1,500 MTok \* $25.00/MTok = $37,500 USD
* 每月成本：$143,000 + $37,500 = $180,500 USD
實際成本可能與這些估算有所不同。這些估算是基於
[批次處理](https://platform.claude.com/docs/zh-TW/about-claude/use-case-guides/content-moderation#consider-batch-processing)
一節中所強調的提示。透過從回應中移除
`explanation`
欄位，可以進一步減少輸出 token。
### 建構強大的提示
要使用 Claude 進行內容審核，Claude 必須了解您應用程式的審核需求。首先撰寫一個能讓您定義審核需求的提示：
```python Python
def moderate_message(message, unsafe_categories):
# 將不安全類別轉換為字串，每個類別各佔一行
unsafe_category_str = "\n".join(unsafe_categories)
# 建構給 Claude 的提示，包含訊息與不安全類別
assessment_prompt = f"""
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
{message}
Unsafe Categories:
{unsafe_category_str}
Respond with ONLY a JSON object, using the format below:
{{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}}
Do not include markdown formatting or code fences in your response."""
# 將請求傳送給 Claude 進行內容審核
response = client.messages.create(
model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens=200,
messages=[{"role": "user", "content": assessment_prompt}],
)
# 解析 Claude 回傳的 JSON 回應
text_block = next(block for block in response.content if block.type == "text")
assessment = json.loads(text_block.text)
# 從評估結果中擷取違規狀態
contains_violation = assessment["violation"]
# 若有違規，取得類別與說明；否則使用空的預設值
violated_categories = assessment.get("categories", []) if contains_violation else []
explanation = assessment.get("explanation") if contains_violation else None
return contains_violation, violated_categories, explanation
# 處理每則留言並列印結果
for comment in user_comments:
print(f"\nComment: {comment}")
violation, violated_categories, explanation = moderate_message(
comment, unsafe_categories
)
if violation:
print(f"Violated Categories: {', '.join(violated_categories)}")
print(f"Explanation: {explanation}")
else:
print("No issues detected.")
```
```typescript TypeScript
// Claude 回傳的 JSON 評估結果的結構
interface ModerationAssessment {
violation: boolean;
categories?: string[];
explanation?: string;
}
async function moderateMessage(
message: string,
unsafeCategories: string[]
): Promise<{ violation: boolean; violatedCategories: string[]; explanation?: string }> {
// 將不安全類別轉換為字串，每個類別各佔一行
const unsafeCategoryStr = unsafeCategories.join("\n");
// 建構給 Claude 的提示，包含訊息與不安全類別
const assessmentPrompt = `
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
${message}
Unsafe Categories:
${unsafeCategoryStr}
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.`;
// 將請求傳送給 Claude 進行內容審核
const response = await client.messages.create({
model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs
max_tokens: 200,
messages: [{ role: "user", content: assessmentPrompt }]
});
// 解析 Claude 回傳的 JSON 回應
const textBlock = response.content.find((block) => block.type === "text");
if (!textBlock) {
throw new Error("Expected a text block in the response");
}
const assessment: ModerationAssessment = JSON.parse(textBlock.text);
// 從評估結果中擷取違規狀態
const containsViolation = assessment.violation;
// 若有違規，取得類別與說明；否則使用空的預設值
const violatedCategories = containsViolation ? (assessment.categories ?? []) : [];
const explanation = containsViolation ? assessment.explanation : undefined;
return { violation: containsViolation, violatedCategories, explanation };
}
// 處理每則留言並印出結果
for (const comment of userComments) {
console.log(`\nComment: ${comment}`);
const { violation, violatedCategories, explanation } = await moderateMessage(
comment,
unsafeCategories
);
if (violation) {
console.log(`Violated Categories: ${violatedCategories.join(", ")}`);
console.log(`Explanation: ${explanation}`);
} else {
console.log("No issues detected.");
}
}
```
```csharp C#
async Task<(bool ContainsViolation, List ViolatedCategories, string? Explanation)> ModerateMessage(
string message,
IReadOnlyList categories
)
{
// 將不安全類別轉換為字串，每個類別各佔一行
var unsafeCategoryText = string.Join("\n", categories);
// 建構給 Claude 的提示，包含訊息與不安全類別
var assessmentPrompt = $$"""
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
{{message}}
Unsafe Categories:
{{unsafeCategoryText}}
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
""";
// 將請求傳送給 Claude 進行內容審核
var response = await client.Messages.Create(
new()
{
Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens = 200,
Messages = [new() { Role = Role.User, Content = assessmentPrompt }],
}
);
// 將第一個內容區塊縮小為文字區塊，然後解析 Claude 的 JSON 回應
if (!response.Content[0].TryPickText(out var textBlock))
{
throw new InvalidOperationException("Expected a text response from Claude.");
}
var assessment = JsonNode.Parse(textBlock.Text)!;
// 從評估結果中擷取違規狀態
var containsViolation = assessment["violation"]!.GetValue();
// 若有違規，取得類別與說明；否則使用空的預設值
List violatedCategories = containsViolation
? assessment["categories"]?.AsArray().Select(category => category!.GetValue()).ToList() ?? []
: [];
var explanation = containsViolation ? assessment["explanation"]?.GetValue() : null;
return (containsViolation, violatedCategories, explanation);
}
// 處理每則留言並列印結果
foreach (var comment in userComments)
{
Console.WriteLine($"\nComment: {comment}");
var (violation, violatedCategories, explanation) = await ModerateMessage(comment, unsafeCategories);
if (violation)
{
Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");
Console.WriteLine($"Explanation: {explanation}");
}
else
{
Console.WriteLine("No issues detected.");
}
}
```
```go Go
func moderateMessage(message string, unsafeCategories []string) (bool, []string, string) {
// 將不安全類別轉換為字串，每個類別各佔一行
unsafeCategoryStr := strings.Join(unsafeCategories, "\n")
// 建構給 Claude 的提示，包含訊息與不安全類別
assessmentPrompt := fmt.Sprintf(`
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
%s
Unsafe Categories:
%s
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)
// 將請求傳送給 Claude 進行內容審核
response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens: 200,
Messages: []anthropic.MessageParam{
anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),
},
})
if err != nil {
log.Fatal(err)
}
// 在讀取文字前，先將第一個內容區塊縮小為文字區塊
textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)
if !ok {
log.Fatalf("expected a text block, got %q", response.Content[0].Type)
}
// 解析 Claude 回傳的 JSON 回應
var assessment struct {
Violation bool `json:"violation"`
Categories []string `json:"categories"`
Explanation string `json:"explanation"`
}
if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {
log.Fatal(err)
}
// 若有違規，回傳類別與說明；否則使用空的預設值
if !assessment.Violation {
return false, nil, ""
}
return true, assessment.Categories, assessment.Explanation
}
// moderateAllComments 會處理每則留言並印出結果。
func moderateAllComments() {
for _, comment := range userComments {
fmt.Printf("\nComment: %s\n", comment)
violation, violatedCategories, explanation := moderateMessage(comment, unsafeCategories)
if violation {
fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))
fmt.Printf("Explanation: %s\n", explanation)
} else {
fmt.Println("No issues detected.")
}
}
}
```
```java Java
record ModerationResult(boolean violation, List violatedCategories, String explanation) {}
ModerationResult moderateMessage(String message, List unsafeCategories)
throws JsonProcessingException {
// 將不安全類別轉換為字串，每個類別各佔一行
String unsafeCategoryStr = String.join("\n", unsafeCategories);
// 建構給 Claude 的提示，包含訊息與不安全類別
String assessmentPrompt = """
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
%s
Unsafe Categories:
%s
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response."""
.formatted(message, unsafeCategoryStr);
// 將請求傳送給 Claude 進行內容審核
Message response = client.messages().create(MessageCreateParams.builder()
.model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs
.maxTokens(200)
.addUserMessage(assessmentPrompt)
.build());
// 解析 Claude 回傳的 JSON 回應
String assessmentJson = response.content().stream()
.flatMap(contentBlock -> contentBlock.text().stream())
.findFirst()
.orElseThrow()
.text();
ObjectMapper mapper = new ObjectMapper();
JsonNode assessment = mapper.readTree(assessmentJson);
// 從評估結果中擷取違規狀態
boolean containsViolation = assessment.required("violation").asBoolean();
// 若有違規，取得類別與說明；否則使用空的預設值
List violatedCategories = containsViolation && assessment.has("categories")
? mapper.convertValue(assessment.get("categories"), new TypeReference>() {})
: List.of();
String explanation = containsViolation && assessment.hasNonNull("explanation")
? assessment.get("explanation").asText()
: null;
return new ModerationResult(containsViolation, violatedCategories, explanation);
}
// 處理每則留言並印出結果
void printModerationResults() throws JsonProcessingException {
for (String comment : userComments) {
IO.println("\nComment: " + comment);
ModerationResult result = moderateMessage(comment, unsafeCategories);
if (result.violation()) {
IO.println("Violated Categories: " + String.join(", ", result.violatedCategories()));
IO.println("Explanation: " + result.explanation());
} else {
IO.println("No issues detected.");
}
}
}
```
```php PHP
$moderateMessage = function (string $message, array $unsafeCategories) use ($client): array {
// 將不安全類別轉換為字串，每個類別各佔一行
$unsafeCategoryStr = implode("\n", $unsafeCategories);
// 建構給 Claude 的提示，包含訊息與不安全類別
$assessmentPrompt = <<{$message}
Unsafe Categories:
{$unsafeCategoryStr}
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
PROMPT;
// 將請求傳送給 Claude 進行內容審核
$response = $client->messages->create(
model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs
maxTokens: 200,
messages: [['role' => 'user', 'content' => $assessmentPrompt]],
);
// 解析 Claude 回傳的 JSON 回應。SDK 會將每個內容區塊
// 解碼為其具體類別，因此需先找到 TextBlock 再讀取文字。
$textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)
?? throw new \RuntimeException('Expected a text block in the response.');
$assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);
// 從評估結果中擷取違規狀態
$containsViolation = $assessment['violation'];
// 若有違規，取得類別與說明；否則使用空的預設值
$violatedCategories = $containsViolation ? ($assessment['categories'] ?? []) : [];
$explanation = $containsViolation ? ($assessment['explanation'] ?? null) : null;
return [$containsViolation, $violatedCategories, $explanation];
};
// 處理每則留言並印出結果
foreach ($userComments as $comment) {
echo "\nComment: {$comment}\n";
[$violation, $violatedCategories, $explanation] = $moderateMessage($comment, $unsafeCategories);
if ($violation) {
echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";
echo "Explanation: {$explanation}\n";
} else {
echo "No issues detected.\n";
}
}
```
```ruby Ruby
def moderate_message(message, unsafe_categories)
# 將不安全類別轉換為字串，每個類別各佔一行
unsafe_category_str = unsafe_categories.join("\n")
# 建構給 Claude 的提示，包含訊息與不安全類別
assessment_prompt = <<~PROMPT.chomp
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
#{message}
Unsafe Categories:
#{unsafe_category_str}
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
PROMPT
# 將請求傳送給 Claude 進行內容審核
response = CLIENT.messages.create(
model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens: 200,
messages: [{role: :user, content: assessment_prompt}]
)
# 解析 Claude 回傳的 JSON 回應
text_block = response.content.find { it.type == :text }
assessment = JSON.parse(text_block.text)
# 從評估結果中擷取違規狀態
contains_violation = assessment["violation"]
# 若有違規，取得類別與說明；否則使用空的預設值
violated_categories = contains_violation ? assessment.fetch("categories", []) : []
explanation = contains_violation ? assessment["explanation"] : nil
[contains_violation, violated_categories, explanation]
end
# 處理每則留言並列印結果
USER_COMMENTS.each do |comment|
puts "\nComment: #{comment}"
violation, violated_categories, explanation = moderate_message(comment, UNSAFE_CATEGORIES)
if violation
puts "Violated Categories: #{violated_categories.join(", ")}"
puts "Explanation: #{explanation}"
else
puts "No issues detected."
end
end
```
在此範例中，`moderate_message` 函式包含一個評估提示，其中包括不安全內容類別以及要評估的訊息。該提示要求 Claude 根據您先前定義的不安全類別，評估該訊息是否應被審核。
接著會解析模型的評估結果，以判斷是否存在違規。如果存在違規，Claude 也會回傳違反的類別清單，以及說明該訊息為何不安全的解釋。
### 評估您的提示
內容審核是一個分類問題。因此，您可以使用[分類 cookbook](https://platform.claude.com/cookbook/capabilities-classification-guide) 中概述的相同技術，來判斷您內容審核系統的準確度。
另一項額外的考量是，與其將內容審核視為二元分類問題，您可以改為建立多個類別來代表不同的風險等級。建立多個風險等級可讓您調整審核的嚴格程度。例如，您可能希望自動封鎖被視為高風險的使用者查詢，而擁有許多中風險查詢的使用者則被標記以供人工審查。
```python Python
def assess_risk_level(message, unsafe_categories):
# 將不安全類別轉換為字串，每個類別各佔一行
unsafe_category_str = "\n".join(unsafe_categories)
# 建構給 Claude 的提示，包含訊息、不安全類別與風險等級定義
assessment_prompt = f"""
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
{message}
Unsafe Categories:
{unsafe_category_str}
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{{
"risk_level": ,
"categories": [Comma-separated list of violated categories],
"explanation":
}}
Do not include markdown formatting or code fences in your response."""
# 將請求傳送給 Claude 進行風險評估
response = client.messages.create(
model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens=200,
messages=[{"role": "user", "content": assessment_prompt}],
)
# 解析 Claude 回傳的 JSON 回應
text_block = next(block for block in response.content if block.type == "text")
assessment = json.loads(text_block.text)
# 從評估結果中擷取風險等級、違規類別與說明
risk_level = assessment["risk_level"]
violated_categories = assessment["categories"]
explanation = assessment.get("explanation")
return risk_level, violated_categories, explanation
# 處理每則留言並列印結果
for comment in user_comments:
print(f"\nComment: {comment}")
risk_level, violated_categories, explanation = assess_risk_level(
comment, unsafe_categories
)
print(f"Risk Level: {risk_level}")
if violated_categories:
print(f"Violated Categories: {', '.join(violated_categories)}")
if explanation:
print(f"Explanation: {explanation}")
```
```typescript TypeScript
// Claude 回傳的 JSON 風險評估結果的結構
interface RiskAssessment {
risk_level: number;
categories: string[];
explanation?: string;
}
async function assessRiskLevel(
message: string,
unsafeCategories: string[]
): Promise<{ riskLevel: number; violatedCategories: string[]; explanation?: string }> {
// 將不安全類別轉換為字串，每個類別各佔一行
const unsafeCategoryStr = unsafeCategories.join("\n");
// 建構給 Claude 的提示，包含訊息、不安全類別與風險等級定義
const assessmentPrompt = `
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
${message}
Unsafe Categories:
${unsafeCategoryStr}
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": ,
"categories": [Comma-separated list of violated categories],
"explanation":
}
Do not include markdown formatting or code fences in your response.`;
// 將請求傳送給 Claude 進行風險評估
const response = await client.messages.create({
model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs
max_tokens: 200,
messages: [{ role: "user", content: assessmentPrompt }]
});
// 解析 Claude 回傳的 JSON 回應
const textBlock = response.content.find((block) => block.type === "text");
if (!textBlock) {
throw new Error("Expected a text block in the response");
}
const assessment: RiskAssessment = JSON.parse(textBlock.text);
// 從評估結果中擷取風險等級、違規類別與說明
const { risk_level: riskLevel, categories: violatedCategories, explanation } = assessment;
return { riskLevel, violatedCategories, explanation };
}
// 處理每則留言並印出結果
for (const comment of userComments) {
console.log(`\nComment: ${comment}`);
const { riskLevel, violatedCategories, explanation } = await assessRiskLevel(
comment,
unsafeCategories
);
console.log(`Risk Level: ${riskLevel}`);
if (violatedCategories.length > 0) {
console.log(`Violated Categories: ${violatedCategories.join(", ")}`);
}
if (explanation) {
console.log(`Explanation: ${explanation}`);
}
}
```
```csharp C#
async Task<(int RiskLevel, List ViolatedCategories, string? Explanation)> AssessRiskLevel(
string message,
IReadOnlyList categories
)
{
// 將不安全類別轉換為字串，每個類別各佔一行
var unsafeCategoryText = string.Join("\n", categories);
// 建構給 Claude 的提示，包含訊息、不安全類別與風險等級定義
var assessmentPrompt = $$"""
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
{{message}}
Unsafe Categories:
{{unsafeCategoryText}}
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": ,
"categories": [Comma-separated list of violated categories],
"explanation":
}
Do not include markdown formatting or code fences in your response.
""";
// 將請求傳送給 Claude 進行風險評估
var response = await client.Messages.Create(
new()
{
Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens = 200,
Messages = [new() { Role = Role.User, Content = assessmentPrompt }],
}
);
// 將第一個內容區塊縮小為文字區塊，然後解析 Claude 的 JSON 回應
if (!response.Content[0].TryPickText(out var textBlock))
{
throw new InvalidOperationException("Expected a text response from Claude.");
}
var assessment = JsonNode.Parse(textBlock.Text)!;
// 從評估結果中擷取風險等級、違規類別與說明
var riskLevel = assessment["risk_level"]!.GetValue();
var violatedCategories = assessment["categories"]!
.AsArray()
.Select(category => category!.GetValue())
.ToList();
var explanation = assessment["explanation"]?.GetValue();
return (riskLevel, violatedCategories, explanation);
}
// 處理每則留言並列印結果
foreach (var comment in userComments)
{
Console.WriteLine($"\nComment: {comment}");
var (riskLevel, violatedCategories, explanation) = await AssessRiskLevel(comment, unsafeCategories);
Console.WriteLine($"Risk Level: {riskLevel}");
if (violatedCategories.Count > 0)
{
Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");
}
if (!string.IsNullOrEmpty(explanation))
{
Console.WriteLine($"Explanation: {explanation}");
}
}
```
```go Go
func assessRiskLevel(message string, unsafeCategories []string) (int, []string, string) {
// 將不安全類別轉換為字串，每個類別各佔一行
unsafeCategoryStr := strings.Join(unsafeCategories, "\n")
// 建構給 Claude 的提示，包含訊息、不安全類別與風險等級定義
assessmentPrompt := fmt.Sprintf(`
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
%s
Unsafe Categories:
%s
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": ,
"categories": [Comma-separated list of violated categories],
"explanation":
}
Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)
// 將請求傳送給 Claude 進行風險評估
response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens: 200,
Messages: []anthropic.MessageParam{
anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),
},
})
if err != nil {
log.Fatal(err)
}
// 在讀取文字前，先將第一個內容區塊縮小為文字區塊
textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)
if !ok {
log.Fatalf("expected a text block, got %q", response.Content[0].Type)
}
// 解析 Claude 回傳的 JSON 回應
var assessment struct {
RiskLevel int `json:"risk_level"`
Categories []string `json:"categories"`
Explanation string `json:"explanation"`
}
if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {
log.Fatal(err)
}
// 回傳評估結果中的風險等級、違規類別與說明
return assessment.RiskLevel, assessment.Categories, assessment.Explanation
}
// assessAllRiskLevels 會處理每則留言並印出結果。
func assessAllRiskLevels() {
for _, comment := range userComments {
fmt.Printf("\nComment: %s\n", comment)
riskLevel, violatedCategories, explanation := assessRiskLevel(comment, unsafeCategories)
fmt.Printf("Risk Level: %d\n", riskLevel)
if len(violatedCategories) > 0 {
fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))
}
if explanation != "" {
fmt.Printf("Explanation: %s\n", explanation)
}
}
}
```
```java Java
record RiskAssessment(int riskLevel, List violatedCategories, String explanation) {}
RiskAssessment assessRiskLevel(String message, List unsafeCategories)
throws JsonProcessingException {
// 將不安全類別轉換為字串，每個類別各佔一行
String unsafeCategoryStr = String.join("\n", unsafeCategories);
// 建構給 Claude 的提示，包含訊息、不安全類別與風險等級定義
String assessmentPrompt = """
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
%s
Unsafe Categories:
%s
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": ,
"categories": [Comma-separated list of violated categories],
"explanation":
}
Do not include markdown formatting or code fences in your response."""
.formatted(message, unsafeCategoryStr);
// 將請求傳送給 Claude 進行風險評估
Message response = client.messages().create(MessageCreateParams.builder()
.model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs
.maxTokens(200)
.addUserMessage(assessmentPrompt)
.build());
// 解析 Claude 回傳的 JSON 回應
String assessmentJson = response.content().stream()
.flatMap(contentBlock -> contentBlock.text().stream())
.findFirst()
.orElseThrow()
.text();
ObjectMapper mapper = new ObjectMapper();
JsonNode assessment = mapper.readTree(assessmentJson);
// 從評估結果中擷取風險等級、違規類別與說明
int riskLevel = assessment.required("risk_level").asInt();
JsonNode categoriesNode = assessment.required("categories");
List violatedCategories = categoriesNode.isNull()
? List.of()
: mapper.convertValue(categoriesNode, new TypeReference>() {});
String explanation = assessment.hasNonNull("explanation")
? assessment.get("explanation").asText()
: null;
return new RiskAssessment(riskLevel, violatedCategories, explanation);
}
// 處理每則留言並印出結果
void printRiskLevels() throws JsonProcessingException {
for (String comment : userComments) {
IO.println("\nComment: " + comment);
RiskAssessment assessment = assessRiskLevel(comment, unsafeCategories);
IO.println("Risk Level: " + assessment.riskLevel());
if (!assessment.violatedCategories().isEmpty()) {
IO.println("Violated Categories: " + String.join(", ", assessment.violatedCategories()));
}
if (assessment.explanation() != null && !assessment.explanation().isEmpty()) {
IO.println("Explanation: " + assessment.explanation());
}
}
}
```
```php PHP
$assessRiskLevel = function (string $message, array $unsafeCategories) use ($client): array {
// 將不安全類別轉換為字串，每個類別各佔一行
$unsafeCategoryStr = implode("\n", $unsafeCategories);
// 建構給 Claude 的提示，包含訊息、不安全類別與風險等級定義
$assessmentPrompt = <<{$message}
Unsafe Categories:
{$unsafeCategoryStr}
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": ,
"categories": [Comma-separated list of violated categories],
"explanation":
}
Do not include markdown formatting or code fences in your response.
PROMPT;
// 將請求傳送給 Claude 進行風險評估
$response = $client->messages->create(
model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs
maxTokens: 200,
messages: [['role' => 'user', 'content' => $assessmentPrompt]],
);
// 解析 Claude 回傳的 JSON 回應。SDK 會將每個內容區塊
// 解碼為其具體類別，因此需先找到 TextBlock 再讀取文字。
$textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)
?? throw new \RuntimeException('Expected a text block in the response.');
$assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);
// 從評估結果中擷取風險等級、違規類別與說明
$riskLevel = $assessment['risk_level'];
$violatedCategories = $assessment['categories'];
$explanation = $assessment['explanation'] ?? null;
return [$riskLevel, $violatedCategories, $explanation];
};
// 處理每則留言並印出結果
foreach ($userComments as $comment) {
echo "\nComment: {$comment}\n";
[$riskLevel, $violatedCategories, $explanation] = $assessRiskLevel($comment, $unsafeCategories);
echo "Risk Level: {$riskLevel}\n";
if ($violatedCategories) {
echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";
}
if ($explanation) {
echo "Explanation: {$explanation}\n";
}
}
```
```ruby Ruby
def assess_risk_level(message, unsafe_categories)
# 將不安全類別轉換為字串，每個類別各佔一行
unsafe_category_str = unsafe_categories.join("\n")
# 建構給 Claude 的提示，包含訊息、不安全類別與風險等級定義
assessment_prompt = <<~PROMPT.chomp
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
#{message}
Unsafe Categories:
#{unsafe_category_str}
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": ,
"categories": [Comma-separated list of violated categories],
"explanation":
}
Do not include markdown formatting or code fences in your response.
PROMPT
# 將請求傳送給 Claude 進行風險評估
response = CLIENT.messages.create(
model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens: 200,
messages: [{role: :user, content: assessment_prompt}]
)
# 解析 Claude 回傳的 JSON 回應
text_block = response.content.find { it.type == :text }
assessment = JSON.parse(text_block.text)
# 從評估結果中擷取風險等級、違規類別與說明
risk_level = assessment["risk_level"]
violated_categories = assessment["categories"]
explanation = assessment["explanation"]
[risk_level, violated_categories, explanation]
end
# 處理每則留言並列印結果
USER_COMMENTS.each do |comment|
puts "\nComment: #{comment}"
risk_level, violated_categories, explanation = assess_risk_level(comment, UNSAFE_CATEGORIES)
puts "Risk Level: #{risk_level}"
puts "Violated Categories: #{violated_categories.join(", ")}" if violated_categories&.any?
puts "Explanation: #{explanation}" if explanation
end
```
此程式碼實作了一個 `assess_risk_level` 函式，使用 Claude 來評估訊息的風險等級。該函式接受一則訊息以及不安全類別作為輸入。
在函式內部，會為 Claude 產生一個提示，其中包括要評估的訊息、不安全類別，以及評估風險等級的具體指示。該提示指示 Claude 以 JSON 物件回應，其中包含風險等級、違反的類別，以及選擇性的解釋。
這種方法透過指派風險等級來實現彈性的內容審核。它可以無縫整合到更大的系統中，以根據評估出的風險等級自動過濾內容或標記留言以供人工審查。例如，執行此程式碼時，留言 `Delete this post now or you better hide. I am coming after you and your family.` 因其危險的威脅而被識別為高風險。相反地，留言 `Stay away from the 5G cellphones!! They are using 5G to control you.` 則被歸類為中風險。
### 部署您的提示
一旦您對解決方案的品質有信心，就可以將其部署到正式環境了。以下是在正式環境中使用內容審核時應遵循的一些最佳實務：
1. **向使用者提供清楚的回饋：** 當使用者輸入因內容審核而被封鎖或回應被標記時，請提供具資訊性且有建設性的回饋，幫助使用者了解其訊息為何被標記，以及如何適當地重新表述。在先前的程式碼範例中，這是透過 Claude 回應中的 `explanation` 欄位來完成的。
2. **分析被審核的內容：** 追蹤您的審核系統所標記的內容類型，以識別趨勢與潛在的改進領域。
3. **持續評估與改進：** 使用精確率（precision）與召回率（recall）追蹤等指標，定期評估您內容審核系統的效能。利用這些資料反覆改進您的審核提示、關鍵字與評估標準。
***
## 提升效能
在複雜的情境中，除了標準的[提示工程技術](https://platform.claude.com/docs/zh-TW/build-with-claude/prompt-engineering/overview)之外，考慮其他策略來提升效能可能會有所幫助。以下是一些進階策略：
### 定義主題並提供範例
除了在提示中列出不安全類別之外，還可以透過提供與每個類別相關的定義與詞句來進一步改進。
```python Python
# 內容審核中被視為不安全的類別及其定義
unsafe_category_definitions = {
"Child Exploitation": "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",
"Conspiracy Theories": """Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.""",
"Hate": """Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.""",
"Indiscriminate Weapons": "Content that enables the creation of weapons of mass destruction.",
"Intellectual Property": "Content that violates the intellectual property rights of third parties.",
"Non-Violent Crimes": "Content that enables, encourages, or excuses the commission of non-violent crimes.",
"Privacy": "Content that contains sensitive, personal information about private individuals.",
"Self-Harm": "Content that enables, encourages, or excuses acts of intentional self-harm.",
"Sex Crimes": "Content that enables, encourages, or excuses the commission of sex crimes.",
"Sexual Content": "Content that contains erotic descriptions or explicit depictions of sex acts.",
"Specialized Advice": """Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.""",
"Violent Crimes": "Content that enables, encourages, or excuses the commission of violent crimes.",
}
def moderate_message_with_definitions(message, unsafe_category_definitions):
# 格式化不安全類別字串，每個類別及其定義各佔一行
unsafe_category_str = "\n".join(
f"{category}: {definition}"
for category, definition in unsafe_category_definitions.items()
)
# 建構給 Claude 的提示，包含訊息與不安全類別
assessment_prompt = f"""Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
{message}
Unsafe Categories and Their Definitions:
{unsafe_category_str}
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}}
Do not include markdown formatting or code fences in your response."""
# 將請求傳送給 Claude 進行內容審核
response = client.messages.create(
model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens=200,
messages=[{"role": "user", "content": assessment_prompt}],
)
# 解析 Claude 回傳的 JSON 回應
text_block = next(block for block in response.content if block.type == "text")
assessment = json.loads(text_block.text)
# 從評估結果中擷取違規狀態
contains_violation = assessment["violation"]
# 若有違規，取得類別與說明；否則使用空的預設值
violated_categories = assessment.get("categories", []) if contains_violation else []
explanation = assessment.get("explanation") if contains_violation else None
return contains_violation, violated_categories, explanation
# 處理每則留言並列印結果
for comment in user_comments:
print(f"\nComment: {comment}")
violation, violated_categories, explanation = moderate_message_with_definitions(
comment, unsafe_category_definitions
)
if violation:
print(f"Violated Categories: {', '.join(violated_categories)}")
print(f"Explanation: {explanation}")
else:
print("No issues detected.")
```
```typescript TypeScript
// Claude 回傳的 JSON 評估結果的結構
interface DefinitionBasedAssessment {
violation: boolean;
categories?: string[];
explanation?: string;
}
// 內容審核中視為不安全的類別及其定義
// （物件鍵會保留插入順序，因此類別會依此順序呈現）
const unsafeCategoryDefinitions: Record = {
"Child Exploitation":
"Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",
"Conspiracy Theories": `Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.`,
Hate: `Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.`,
"Indiscriminate Weapons":
"Content that enables the creation of weapons of mass destruction.",
"Intellectual Property":
"Content that violates the intellectual property rights of third parties.",
"Non-Violent Crimes":
"Content that enables, encourages, or excuses the commission of non-violent crimes.",
Privacy: "Content that contains sensitive, personal information about private individuals.",
"Self-Harm": "Content that enables, encourages, or excuses acts of intentional self-harm.",
"Sex Crimes": "Content that enables, encourages, or excuses the commission of sex crimes.",
"Sexual Content":
"Content that contains erotic descriptions or explicit depictions of sex acts.",
"Specialized Advice": `Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.`,
"Violent Crimes":
"Content that enables, encourages, or excuses the commission of violent crimes."
};
async function moderateMessageWithDefinitions(
message: string,
unsafeCategoryDefinitions: Record
): Promise<{ violation: boolean; violatedCategories: string[]; explanation?: string }> {
// 格式化不安全類別字串，每個類別及其定義各佔一行
const unsafeCategoryStr = Object.entries(unsafeCategoryDefinitions)
.map(([category, definition]) => `${category}: ${definition}`)
.join("\n");
// 建構給 Claude 的提示，包含訊息與不安全類別
const assessmentPrompt = `Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
${message}
Unsafe Categories and Their Definitions:
${unsafeCategoryStr}
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.`;
// 將請求傳送給 Claude 進行內容審核
const response = await client.messages.create({
model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs
max_tokens: 200,
messages: [{ role: "user", content: assessmentPrompt }]
});
// 解析 Claude 回傳的 JSON 回應
const textBlock = response.content.find((block) => block.type === "text");
if (!textBlock) {
throw new Error("Expected a text block in the response");
}
const assessment: DefinitionBasedAssessment = JSON.parse(textBlock.text);
// 從評估結果中擷取違規狀態
const containsViolation = assessment.violation;
// 若有違規，取得類別與說明；否則使用空的預設值
const violatedCategories = containsViolation ? (assessment.categories ?? []) : [];
const explanation = containsViolation ? assessment.explanation : undefined;
return { violation: containsViolation, violatedCategories, explanation };
}
// 處理每則留言並印出結果
for (const comment of userComments) {
console.log(`\nComment: ${comment}`);
const { violation, violatedCategories, explanation } = await moderateMessageWithDefinitions(
comment,
unsafeCategoryDefinitions
);
if (violation) {
console.log(`Violated Categories: ${violatedCategories.join(", ")}`);
console.log(`Explanation: ${explanation}`);
} else {
console.log("No issues detected.");
}
}
```
```csharp C#
// 內容審核中視為不安全的類別及其定義。
// 這些項目保持插入順序，因此產生的提示會完全依照
// 此順序列出類別。
(string Category, string Definition)[] unsafeCategoryDefinitions =
[
(
"Child Exploitation",
"Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."
),
(
"Conspiracy Theories",
"""
Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.
"""
),
(
"Hate",
"""
Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.
"""
),
("Indiscriminate Weapons", "Content that enables the creation of weapons of mass destruction."),
("Intellectual Property", "Content that violates the intellectual property rights of third parties."),
("Non-Violent Crimes", "Content that enables, encourages, or excuses the commission of non-violent crimes."),
("Privacy", "Content that contains sensitive, personal information about private individuals."),
("Self-Harm", "Content that enables, encourages, or excuses acts of intentional self-harm."),
("Sex Crimes", "Content that enables, encourages, or excuses the commission of sex crimes."),
("Sexual Content", "Content that contains erotic descriptions or explicit depictions of sex acts."),
(
"Specialized Advice",
"""
Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.
"""
),
("Violent Crimes", "Content that enables, encourages, or excuses the commission of violent crimes."),
];
async Task<(bool ContainsViolation, List ViolatedCategories, string? Explanation)> ModerateMessageWithDefinitions(
string message,
IReadOnlyList<(string Category, string Definition)> categoryDefinitions
)
{
// 格式化不安全類別字串，每個類別及其定義各佔一行
var unsafeCategoryText = string.Join(
"\n",
categoryDefinitions.Select(entry => $"{entry.Category}: {entry.Definition}")
);
// 建構給 Claude 的提示，包含訊息與不安全類別
var assessmentPrompt = $$"""
Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
{{message}}
Unsafe Categories and Their Definitions:
{{unsafeCategoryText}}
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
""";
// 將請求傳送給 Claude 進行內容審核
var response = await client.Messages.Create(
new()
{
Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens = 200,
Messages = [new() { Role = Role.User, Content = assessmentPrompt }],
}
);
// 將第一個內容區塊縮小為文字區塊，然後解析 Claude 的 JSON 回應
if (!response.Content[0].TryPickText(out var textBlock))
{
throw new InvalidOperationException("Expected a text response from Claude.");
}
var assessment = JsonNode.Parse(textBlock.Text)!;
// 從評估結果中擷取違規狀態
var containsViolation = assessment["violation"]!.GetValue();
// 若有違規，取得類別與說明；否則使用空的預設值
List violatedCategories = containsViolation
? assessment["categories"]?.AsArray().Select(category => category!.GetValue()).ToList() ?? []
: [];
var explanation = containsViolation ? assessment["explanation"]?.GetValue() : null;
return (containsViolation, violatedCategories, explanation);
}
// 處理每則留言並列印結果
foreach (var comment in userComments)
{
Console.WriteLine($"\nComment: {comment}");
var (violation, violatedCategories, explanation) = await ModerateMessageWithDefinitions(
comment,
unsafeCategoryDefinitions
);
if (violation)
{
Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");
Console.WriteLine($"Explanation: {explanation}");
}
else
{
Console.WriteLine("No issues detected.");
}
}
```
```go Go
// 內容審核中視為不安全的類別及其定義。
// 使用類別／定義配對的 slice（而非 map）可讓呈現
// 順序保持穩定；Go 的 map 會以隨機順序迭代。
type categoryDefinition struct {
category string
definition string
}
var unsafeCategoryDefinitions = []categoryDefinition{
{"Child Exploitation", "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."},
{"Conspiracy Theories", `Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.`},
{"Hate", `Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.`},
{"Indiscriminate Weapons", "Content that enables the creation of weapons of mass destruction."},
{"Intellectual Property", "Content that violates the intellectual property rights of third parties."},
{"Non-Violent Crimes", "Content that enables, encourages, or excuses the commission of non-violent crimes."},
{"Privacy", "Content that contains sensitive, personal information about private individuals."},
{"Self-Harm", "Content that enables, encourages, or excuses acts of intentional self-harm."},
{"Sex Crimes", "Content that enables, encourages, or excuses the commission of sex crimes."},
{"Sexual Content", "Content that contains erotic descriptions or explicit depictions of sex acts."},
{"Specialized Advice", `Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.`},
{"Violent Crimes", "Content that enables, encourages, or excuses the commission of violent crimes."},
}
func moderateMessageWithDefinitions(message string, unsafeCategoryDefinitions []categoryDefinition) (bool, []string, string) {
// 格式化不安全類別字串，每個類別及其定義各佔一行
categoryLines := make([]string, len(unsafeCategoryDefinitions))
for i, entry := range unsafeCategoryDefinitions {
categoryLines[i] = fmt.Sprintf("%s: %s", entry.category, entry.definition)
}
unsafeCategoryStr := strings.Join(categoryLines, "\n")
// 建構給 Claude 的提示，包含訊息與不安全類別
assessmentPrompt := fmt.Sprintf(`Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
%s
Unsafe Categories and Their Definitions:
%s
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)
// 將請求傳送給 Claude 進行內容審核
response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens: 200,
Messages: []anthropic.MessageParam{
anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),
},
})
if err != nil {
log.Fatal(err)
}
// 在讀取文字前，先將第一個內容區塊縮小為文字區塊
textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)
if !ok {
log.Fatalf("expected a text block, got %q", response.Content[0].Type)
}
// 解析 Claude 回傳的 JSON 回應
var assessment struct {
Violation bool `json:"violation"`
Categories []string `json:"categories"`
Explanation string `json:"explanation"`
}
if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {
log.Fatal(err)
}
// 若有違規，回傳類別與說明；否則使用空的預設值
if !assessment.Violation {
return false, nil, ""
}
return true, assessment.Categories, assessment.Explanation
}
// moderateAllCommentsWithDefinitions 會處理每則留言並印出結果。
func moderateAllCommentsWithDefinitions() {
for _, comment := range userComments {
fmt.Printf("\nComment: %s\n", comment)
violation, violatedCategories, explanation := moderateMessageWithDefinitions(comment, unsafeCategoryDefinitions)
if violation {
fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))
fmt.Printf("Explanation: %s\n", explanation)
} else {
fmt.Println("No issues detected.")
}
}
}
```
```java Java
// 內容審核中被視為不安全的類別及其定義
record CategoryDefinition(String category, String definition) {}
final List unsafeCategoryDefinitions = List.of(
new CategoryDefinition(
"Child Exploitation",
"Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."),
new CategoryDefinition(
"Conspiracy Theories",
"""
Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts."""),
new CategoryDefinition(
"Hate",
"""
Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes."""),
new CategoryDefinition(
"Indiscriminate Weapons",
"Content that enables the creation of weapons of mass destruction."),
new CategoryDefinition(
"Intellectual Property",
"Content that violates the intellectual property rights of third parties."),
new CategoryDefinition(
"Non-Violent Crimes",
"Content that enables, encourages, or excuses the commission of non-violent crimes."),
new CategoryDefinition(
"Privacy",
"Content that contains sensitive, personal information about private individuals."),
new CategoryDefinition(
"Self-Harm",
"Content that enables, encourages, or excuses acts of intentional self-harm."),
new CategoryDefinition(
"Sex Crimes",
"Content that enables, encourages, or excuses the commission of sex crimes."),
new CategoryDefinition(
"Sexual Content",
"Content that contains erotic descriptions or explicit depictions of sex acts."),
new CategoryDefinition(
"Specialized Advice",
"""
Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning."""),
new CategoryDefinition(
"Violent Crimes",
"Content that enables, encourages, or excuses the commission of violent crimes."));
record ModerationDecision(boolean violation, List violatedCategories, String explanation) {}
ModerationDecision moderateMessageWithDefinitions(
String message, List unsafeCategoryDefinitions)
throws JsonProcessingException {
// 格式化不安全類別字串，每個類別及其定義各佔一行
String unsafeCategoryStr = unsafeCategoryDefinitions.stream()
.map(categoryDefinition ->
categoryDefinition.category() + ": " + categoryDefinition.definition())
.collect(Collectors.joining("\n"));
// 建構給 Claude 的提示，包含訊息與不安全類別
String assessmentPrompt = """
Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
%s
Unsafe Categories and Their Definitions:
%s
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response."""
.formatted(message, unsafeCategoryStr);
// 將請求傳送給 Claude 進行內容審核
Message response = client.messages().create(MessageCreateParams.builder()
.model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs
.maxTokens(200)
.addUserMessage(assessmentPrompt)
.build());
// 解析 Claude 回傳的 JSON 回應
String assessmentJson = response.content().stream()
.flatMap(contentBlock -> contentBlock.text().stream())
.findFirst()
.orElseThrow()
.text();
ObjectMapper mapper = new ObjectMapper();
JsonNode assessment = mapper.readTree(assessmentJson);
// 從評估結果中擷取違規狀態
boolean containsViolation = assessment.required("violation").asBoolean();
// 若有違規，取得類別與說明；否則使用空的預設值
List violatedCategories = containsViolation && assessment.has("categories")
? mapper.convertValue(assessment.get("categories"), new TypeReference>() {})
: List.of();
String explanation = containsViolation && assessment.hasNonNull("explanation")
? assessment.get("explanation").asText()
: null;
return new ModerationDecision(containsViolation, violatedCategories, explanation);
}
// 處理每則留言並印出結果
void printModerationResultsWithDefinitions() throws JsonProcessingException {
for (String comment : userComments) {
IO.println("\nComment: " + comment);
ModerationDecision result = moderateMessageWithDefinitions(comment, unsafeCategoryDefinitions);
if (result.violation()) {
IO.println("Violated Categories: " + String.join(", ", result.violatedCategories()));
IO.println("Explanation: " + result.explanation());
} else {
IO.println("No issues detected.");
}
}
}
```
```php PHP
// 內容審核中被視為不安全的類別及其定義
$unsafeCategoryDefinitions = [
'Child Exploitation' => 'Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.',
'Conspiracy Theories' => 'Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.',
'Hate' => 'Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.',
'Indiscriminate Weapons' => 'Content that enables the creation of weapons of mass destruction.',
'Intellectual Property' => 'Content that violates the intellectual property rights of third parties.',
'Non-Violent Crimes' => 'Content that enables, encourages, or excuses the commission of non-violent crimes.',
'Privacy' => 'Content that contains sensitive, personal information about private individuals.',
'Self-Harm' => 'Content that enables, encourages, or excuses acts of intentional self-harm.',
'Sex Crimes' => 'Content that enables, encourages, or excuses the commission of sex crimes.',
'Sexual Content' => 'Content that contains erotic descriptions or explicit depictions of sex acts.',
'Specialized Advice' => 'Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.',
'Violent Crimes' => 'Content that enables, encourages, or excuses the commission of violent crimes.',
];
$moderateMessageWithDefinitions = function (string $message, array $unsafeCategoryDefinitions) use ($client): array {
// 格式化不安全類別字串，每個類別及其定義各佔一行
$categoryLines = [];
foreach ($unsafeCategoryDefinitions as $category => $definition) {
$categoryLines[] = "{$category}: {$definition}";
}
$unsafeCategoryStr = implode("\n", $categoryLines);
// 建構給 Claude 的提示，包含訊息與不安全類別
$assessmentPrompt = <<{$message}
Unsafe Categories and Their Definitions:
{$unsafeCategoryStr}
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
PROMPT;
// 將請求傳送給 Claude 進行內容審核
$response = $client->messages->create(
model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs
maxTokens: 200,
messages: [['role' => 'user', 'content' => $assessmentPrompt]],
);
// 解析 Claude 回傳的 JSON 回應。SDK 會將每個內容區塊
// 解碼為其具體類別，因此需先找到 TextBlock 再讀取文字。
$textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)
?? throw new \RuntimeException('Expected a text block in the response.');
$assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);
// 從評估結果中擷取違規狀態
$containsViolation = $assessment['violation'];
// 若有違規，取得類別與說明；否則使用空的預設值
$violatedCategories = $containsViolation ? ($assessment['categories'] ?? []) : [];
$explanation = $containsViolation ? ($assessment['explanation'] ?? null) : null;
return [$containsViolation, $violatedCategories, $explanation];
};
// 處理每則留言並印出結果
foreach ($userComments as $comment) {
echo "\nComment: {$comment}\n";
[$violation, $violatedCategories, $explanation] = $moderateMessageWithDefinitions($comment, $unsafeCategoryDefinitions);
if ($violation) {
echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";
echo "Explanation: {$explanation}\n";
} else {
echo "No issues detected.\n";
}
}
```
```ruby Ruby
# 內容審核中視為不安全的類別及其定義
UNSAFE_CATEGORY_DEFINITIONS = {
"Child Exploitation" => "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",
"Conspiracy Theories" => "Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.",
"Hate" => "Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.",
"Indiscriminate Weapons" => "Content that enables the creation of weapons of mass destruction.",
"Intellectual Property" => "Content that violates the intellectual property rights of third parties.",
"Non-Violent Crimes" => "Content that enables, encourages, or excuses the commission of non-violent crimes.",
"Privacy" => "Content that contains sensitive, personal information about private individuals.",
"Self-Harm" => "Content that enables, encourages, or excuses acts of intentional self-harm.",
"Sex Crimes" => "Content that enables, encourages, or excuses the commission of sex crimes.",
"Sexual Content" => "Content that contains erotic descriptions or explicit depictions of sex acts.",
"Specialized Advice" => "Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.",
"Violent Crimes" => "Content that enables, encourages, or excuses the commission of violent crimes."
}
def moderate_message_with_definitions(message, unsafe_category_definitions)
# 格式化不安全類別字串，每個類別及其定義各佔一行
unsafe_category_str = unsafe_category_definitions
.map { |category, definition| "#{category}: #{definition}" }
.join("\n")
# 建構給 Claude 的提示，包含訊息與不安全類別
assessment_prompt = <<~PROMPT.chomp
Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
#{message}
Unsafe Categories and Their Definitions:
#{unsafe_category_str}
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": ,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
PROMPT
# 將請求傳送給 Claude 進行內容審核
response = CLIENT.messages.create(
model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens: 200,
messages: [{role: :user, content: assessment_prompt}]
)
# 解析 Claude 回傳的 JSON 回應
text_block = response.content.find { it.type == :text }
assessment = JSON.parse(text_block.text)
# 從評估結果中擷取違規狀態
contains_violation = assessment["violation"]
# 若有違規，取得類別與說明；否則使用空的預設值
violated_categories = contains_violation ? assessment.fetch("categories", []) : []
explanation = contains_violation ? assessment["explanation"] : nil
[contains_violation, violated_categories, explanation]
end
# 處理每則留言並列印結果
USER_COMMENTS.each do |comment|
puts "\nComment: #{comment}"
violation, violated_categories, explanation = moderate_message_with_definitions(comment, UNSAFE_CATEGORY_DEFINITIONS)
if violation
puts "Violated Categories: #{violated_categories.join(", ")}"
puts "Explanation: #{explanation}"
else
puts "No issues detected."
end
end
```
`moderate_message_with_definitions` 函式擴充了先前的 `moderate_message` 函式，允許每個不安全類別與詳細的定義配對。在程式碼中，這是透過將原始函式中的 `unsafe_categories` 集合替換為 `unsafe_category_definitions` 對映來實現的。此對映將每個不安全類別與其對應的定義配對。類別名稱及其定義都會包含在提示中。
值得注意的是，`Specialized Advice` 類別的定義現在明確指出了應被禁止的財務建議類型。因此，先前通過 `moderate_message` 評估的留言 `It is a great time to invest in gold!`，現在會觸發違規。
### 考慮批次處理
在不需要即時審核的情況下，為了降低成本，可以考慮以批次方式審核訊息。在提示的上下文中包含多則訊息，並要求 Claude 評估哪些訊息應被審核。
```python Python
def batch_moderate_messages(messages, unsafe_categories):
# 將不安全類別轉換為字串，每個類別各佔一行
unsafe_category_str = "\n".join(unsafe_categories)
# 格式化訊息字串，每則訊息以類 XML 標籤包裹並賦予 ID
messages_str = "\n".join(
[f"{msg}" for idx, msg in enumerate(messages)]
)
# 建構給 Claude 的提示，包含訊息與不安全類別
assessment_prompt = f"""Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
{messages_str}
Unsafe Categories:
{unsafe_category_str}
Respond with ONLY a JSON object, using the format below:
{{
"violations": [
{{
"id": ,
"categories": [list of violated categories],
"explanation":
}}
]
}}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response."""
# 將請求傳送給 Claude 進行內容審核
response = client.messages.create(
model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens=2048, # Increased max token count to handle batches
messages=[{"role": "user", "content": assessment_prompt}],
)
# 解析 Claude 回傳的 JSON 回應
text_block = next(block for block in response.content if block.type == "text")
assessment = json.loads(text_block.text)
return assessment
# 批次處理留言並取得回應
response_obj = batch_moderate_messages(user_comments, unsafe_categories)
# 列印每個偵測到的違規結果
for violation in response_obj["violations"]:
print(f"""Comment: {user_comments[violation["id"]]}
Violated Categories: {", ".join(violation["categories"])}
Explanation: {violation["explanation"]}
""")
```
```typescript TypeScript
// Claude 回傳的 JSON 批次評估結果的結構
interface BatchAssessment {
violations: {
id: number;
categories: string[];
explanation: string;
}[];
}
async function batchModerateMessages(
messages: string[],
unsafeCategories: string[]
): Promise {
// 將不安全類別轉換為字串，每個類別各佔一行
const unsafeCategoryStr = unsafeCategories.join("\n");
// 格式化訊息字串，每則訊息以類 XML 標籤包裹並賦予 ID
const messagesStr = messages
.map((msg, idx) => `${msg}`)
.join("\n");
// 建構給 Claude 的提示，包含訊息與不安全類別
const assessmentPrompt = `Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
${messagesStr}
Unsafe Categories:
${unsafeCategoryStr}
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": ,
"categories": [list of violated categories],
"explanation":
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response.`;
// 將請求傳送給 Claude 進行內容審核
const response = await client.messages.create({
model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs
max_tokens: 2048, // Increased max token count to handle batches
messages: [{ role: "user", content: assessmentPrompt }]
});
// 解析 Claude 回傳的 JSON 回應
const textBlock = response.content.find((block) => block.type === "text");
if (!textBlock) {
throw new Error("Expected a text block in the response");
}
const assessment: BatchAssessment = JSON.parse(textBlock.text);
return assessment;
}
// 處理這批留言並取得回應
const batchAssessment = await batchModerateMessages(userComments, unsafeCategories);
// 印出每個偵測到的違規結果
for (const violation of batchAssessment.violations) {
console.log(`Comment: ${userComments[violation.id]}
Violated Categories: ${violation.categories.join(", ")}
Explanation: ${violation.explanation}
`);
}
```
```csharp C#
async Task BatchModerateMessages(IReadOnlyList messages, IReadOnlyList categories)
{
// 將不安全類別轉換為字串，每個類別各佔一行
var unsafeCategoryText = string.Join("\n", categories);
// 格式化訊息字串，每則訊息以類 XML 標籤包裹並賦予 ID
var messagesText = string.Join(
"\n",
messages.Select((message, index) => $"{message}")
);
// 建構給 Claude 的提示，包含訊息與不安全類別
var assessmentPrompt = $$"""
Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
{{messagesText}}
Unsafe Categories:
{{unsafeCategoryText}}
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": ,
"categories": [list of violated categories],
"explanation":
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response.
""";
// 將請求傳送給 Claude 進行內容審核
var response = await client.Messages.Create(
new()
{
Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens = 2048, // Increased max token count to handle batches
Messages = [new() { Role = Role.User, Content = assessmentPrompt }],
}
);
// 將第一個內容區塊縮小為文字區塊，然後解析 Claude 的 JSON 回應
if (!response.Content[0].TryPickText(out var textBlock))
{
throw new InvalidOperationException("Expected a text response from Claude.");
}
return JsonNode.Parse(textBlock.Text)!;
}
// 處理這批留言並取得回應
var moderationResults = await BatchModerateMessages(userComments, unsafeCategories);
// 列印每個偵測到的違規結果
foreach (var violation in moderationResults["violations"]!.AsArray())
{
var flaggedComment = userComments[violation!["id"]!.GetValue()];
var violatedCategories = string.Join(
", ",
violation["categories"]!.AsArray().Select(category => category!.GetValue())
);
var explanation = violation["explanation"]!.GetValue();
Console.WriteLine($"""
Comment: {flaggedComment}
Violated Categories: {violatedCategories}
Explanation: {explanation}
""");
}
```
```go Go
// batchViolation 是 Claude 回傳的 "violations" 陣列中的一個項目：包含
// 違規訊息的索引、其違反的類別以及原因。
type batchViolation struct {
ID int `json:"id"`
Categories []string `json:"categories"`
Explanation string `json:"explanation"`
}
func batchModerateMessages(messages []string, unsafeCategories []string) []batchViolation {
// 將不安全類別轉換為字串，每個類別各佔一行
unsafeCategoryStr := strings.Join(unsafeCategories, "\n")
// 格式化訊息字串，每則訊息以類 XML 標籤包裹並賦予 ID
messageLines := make([]string, len(messages))
for i, message := range messages {
messageLines[i] = fmt.Sprintf("%s", i, message)
}
messagesStr := strings.Join(messageLines, "\n")
// 建構給 Claude 的提示，包含訊息與不安全類別
assessmentPrompt := fmt.Sprintf(`Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
%s
Unsafe Categories:
%s
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": ,
"categories": [list of violated categories],
"explanation":
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response.`, messagesStr, unsafeCategoryStr)
// 將請求傳送給 Claude 進行內容審核
response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens: 2048, // Increased max token count to handle batches
Messages: []anthropic.MessageParam{
anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),
},
})
if err != nil {
log.Fatal(err)
}
// 在讀取文字前，先將第一個內容區塊縮小為文字區塊
textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)
if !ok {
log.Fatalf("expected a text block, got %q", response.Content[0].Type)
}
// 解析 Claude 回傳的 JSON 回應
var assessment struct {
Violations []batchViolation `json:"violations"`
}
if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {
log.Fatal(err)
}
return assessment.Violations
}
// moderateAllCommentsAsBatch 會在單一請求中審核整批留言，
// 並印出每個偵測到的違規結果。
func moderateAllCommentsAsBatch() {
// 處理整批留言並取得回應
violations := batchModerateMessages(userComments, unsafeCategories)
// 印出每個偵測到的違規結果
for _, violation := range violations {
fmt.Printf(`Comment: %s
Violated Categories: %s
Explanation: %s
`, userComments[violation.ID], strings.Join(violation.Categories, ", "), violation.Explanation)
}
}
```
```java Java
JsonNode batchModerateMessages(List messages, List unsafeCategories)
throws JsonProcessingException {
// 將不安全類別轉換為字串，每個類別各佔一行
String unsafeCategoryStr = String.join("\n", unsafeCategories);
// 格式化訊息字串，每則訊息以類 XML 標籤包裹並賦予 ID
String messagesStr = IntStream.range(0, messages.size())
.mapToObj(idx -> "%s".formatted(idx, messages.get(idx)))
.collect(Collectors.joining("\n"));
// 建構給 Claude 的提示，包含訊息與不安全類別
String assessmentPrompt = """
Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
%s
Unsafe Categories:
%s
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": ,
"categories": [list of violated categories],
"explanation":
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response."""
.formatted(messagesStr, unsafeCategoryStr);
// 將請求傳送給 Claude 進行內容審核
Message response = client.messages().create(MessageCreateParams.builder()
.model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs
.maxTokens(2048) // Increased max token count to handle batches
.addUserMessage(assessmentPrompt)
.build());
// 解析 Claude 回傳的 JSON 回應
String assessmentJson = response.content().stream()
.flatMap(contentBlock -> contentBlock.text().stream())
.findFirst()
.orElseThrow()
.text();
return new ObjectMapper().readTree(assessmentJson);
}
// 批次處理留言，並印出每個偵測到的違規結果
void printBatchViolations() throws JsonProcessingException {
JsonNode response = batchModerateMessages(userComments, unsafeCategories);
ObjectMapper mapper = new ObjectMapper();
for (JsonNode violation : response.required("violations")) {
List violatedCategories =
mapper.convertValue(violation.required("categories"), new TypeReference>() {});
IO.println("""
Comment: %s
Violated Categories: %s
Explanation: %s
""".formatted(
userComments.get(violation.required("id").asInt()),
String.join(", ", violatedCategories),
violation.required("explanation").asText()));
}
}
```
```php PHP
$batchModerateMessages = function (array $messages, array $unsafeCategories) use ($client): array {
// 將不安全類別轉換為字串，每個類別各佔一行
$unsafeCategoryStr = implode("\n", $unsafeCategories);
// 格式化訊息字串，每則訊息以類 XML 標籤包裹並賦予 ID
$messageLines = [];
foreach ($messages as $idx => $msg) {
$messageLines[] = "{$msg}";
}
$messagesStr = implode("\n", $messageLines);
// 建構給 Claude 的提示，包含訊息與不安全類別
$assessmentPrompt = <<
{$messagesStr}
Unsafe Categories:
{$unsafeCategoryStr}
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": ,
"categories": [list of violated categories],
"explanation":
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response.
PROMPT;
// 將請求傳送給 Claude 進行內容審核
$response = $client->messages->create(
model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs
maxTokens: 2048, // Increased max token count to handle batches
messages: [['role' => 'user', 'content' => $assessmentPrompt]],
);
// 解析 Claude 回傳的 JSON 回應。SDK 會將每個內容區塊
// 解碼為其具體類別，因此需先找到 TextBlock 再讀取文字。
$textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)
?? throw new \RuntimeException('Expected a text block in the response.');
return json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);
};
// 批次處理留言並取得回應
$responseObj = $batchModerateMessages($userComments, $unsafeCategories);
// 印出每個偵測到的違規結果
foreach ($responseObj['violations'] as $violation) {
echo "Comment: {$userComments[$violation['id']]}\n";
echo 'Violated Categories: ' . implode(', ', $violation['categories']) . "\n";
echo "Explanation: {$violation['explanation']}\n\n";
}
```
```ruby Ruby
def batch_moderate_messages(messages, unsafe_categories)
# 將不安全類別轉換為字串，每個類別各佔一行
unsafe_category_str = unsafe_categories.join("\n")
# 格式化訊息字串，每則訊息以類 XML 標籤包裹並賦予 ID
messages_str = messages
.map.with_index { |message, index| "#{message}" }
.join("\n")
# 建構給 Claude 的提示，包含訊息與不安全類別
assessment_prompt = <<~PROMPT.chomp
Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
#{messages_str}
Unsafe Categories:
#{unsafe_category_str}
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": ,
"categories": [list of violated categories],
"explanation":
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response.
PROMPT
# 將請求傳送給 Claude 進行內容審核
response = CLIENT.messages.create(
model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens: 2048, # Increased max token count to handle batches
messages: [{role: :user, content: assessment_prompt}]
)
# 解析 Claude 回傳的 JSON 回應
text_block = response.content.find { it.type == :text }
JSON.parse(text_block.text)
end
# 批次處理留言並取得回應
response_obj = batch_moderate_messages(USER_COMMENTS, UNSAFE_CATEGORIES)
# 列印每個偵測到的違規結果
response_obj["violations"].each do |violation|
puts <<~RESULT
Comment: #{USER_COMMENTS[violation["id"]]}
Violated Categories: #{violation["categories"].join(", ")}
Explanation: #{violation["explanation"]}
RESULT
end
```
在此範例中，`batch_moderate_messages` 函式透過單次 Claude API 呼叫來處理整批訊息的審核。 在函式內部，會建立一個提示，其中包括要評估的訊息清單以及不安全內容類別。該提示指示 Claude 回傳一個 JSON 物件，列出所有包含違規的訊息。回應中的每則訊息都以其 `id` 識別，該 `id` 對應於訊息在批次中的位置。 請記住，為您的特定需求找到最佳批次大小可能需要一些實驗。雖然較大的批次大小可以降低成本，但也可能導致品質略微下降。此外，您可能需要增加 Claude API 呼叫中的 `max_tokens` 參數，以容納較長的回應。有關您所選模型可輸出的最大 token 數的詳細資訊，請參閱[模型比較表](https://platform.claude.com/docs/zh-TW/models/overview#latest-models-comparison)。
查看如何使用 Claude 進行內容審核的完整程式碼實作範例。
探索用於審核與 Claude 互動的防護技術。