Moderating uploads

The basic setup

Check the image in your upload handler, on your server:

import express from 'express';
import multer from 'multer';
import { SafetyClient, isSafe } from '@chidakashi-ai/safety';

const upload = multer({ limits: { fileSize: 20 * 1024 * 1024 } });
const client = new SafetyClient(creds);

app.post('/api/upload', upload.single('file'), async (req, res) => {
  const result = await client.checkImage(req.file.buffer, 'uploads', { showReason: true });
  if (!isSafe(result)) {
    return res.status(422).json({ error: 'rejected', reason: result.reason });
  }
  res.json({ url: await store(req.file.buffer) });
});

Your web page sends the file to your endpoint. Never call Kavach from the browser: the API key would be visible to anyone.

Handle unknown

An image can come back unknown when it's too dark, cropped or blurry to judge. Sending those to a person is usually the right call:

switch (result.verdict) {
  case 'pass':    return store(buffer);
  case 'block':   return reject(result.reason);
  case 'unknown': return sendToHumanReview(buffer);
}

Write rules about what is visible

A check only sees the image. Write rules about what can be seen in it:

await policies.save('uploads', `
  1) No nudity or sexually explicit imagery.
  2) No weapons held or aimed, unless holstered on a uniformed guard.
  3) No graphic injury or blood.
  4) Classical and fine-art nudity in a gallery or museum is allowed.
`);

Limits

JPEG, PNG, GIF, BMP or WebP, up to 20 MB. Image checks take longer than text checks.