Connecting via the API

To offer customers an alternative to crawling, we have made several endpoints available for sharing information with our platform from external sources, or reading from our platform. These endpoints are primarily for reading, writing, updating and deleting so-called Documents.

Documents

Within the platform, Documents are the sources from which we generate answers. All documents for a customer together are what we call the knowledge base. With the API, an external party can influence this knowledge base and therefore the quality of the generated answers.

Before you start

The base URL, authentication with a Bearer token and the required headers are described on Working with the API. For the actions on this page, your token needs the permissions documents:create, documents:read, documents:update and documents:delete.

Which source do your documents land in?

Documents you send through the API land in an API source in your knowledge base. When you create a token, you pick that source under Store documents in source; leave it empty and the environment's default API source is used. That way every system you connect can write into its own source, with its own segments and metadata settings. In the API, every document carries the read-only field source_id.

An API source has allowed URLs: one or more patterns, such as https://www.example.com/*. If you send a document whose URL matches none of the patterns, the API rejects it with a 400 error saying the URL is not allowed for this source. That keeps a test environment that accidentally points at production from filling your knowledge base. A new source starts at * (everything allowed); you adjust the patterns in the source's settings in the app, see Your knowledge base in the app.

You list the sources themselves with GET https://{klant}.vragen.ai/api/v1/sources (with the documents:read permission): per source its id, name, type and URL patterns. With GET https://{klant}.vragen.ai/api/v1/documents?filter[source_id]={id} you fetch only the documents of one source.

API documentation

General information and principles

The API follows the JSON:API standard on top of REST; the base URL and the required headers are described on Working with the API.

In responses you will also find the property content_cleaned. This contains the stored content after it has passed through our filter and parser. This lets you validate whether the submitted information is processed correctly.

Document schema

When you add documents to the knowledge base, you can provide the following information:

Basic fields

`external_reference`

Type: string (required)

A reference to the document within your own system. Use this field to link documents in the knowledge base to records in your own database or CMS.

`url`

Type: string (required)

The unique source URL of the document. This is how we make each document unique and distinguishable within the knowledge base. Make sure every URL is unique per document, and that it matches the source's allowed URLs (see above); otherwise you get a 400 error back.

`mime_type`

Type: string (required)

The file type of the source. We currently support the following types:

text/html for HTML documents
application/pdf for PDF files

`title`

Type: string (optional, but recommended)

The title of the document. It is used for display in search results and helps users identify the document.

Automatic extraction: If you do not provide a title, we automatically extract the title from the document itself (for example from an <h1> tag in HTML documents).

`content`

Type: string (optional)

The full text content of the document. This content is indexed and forms the basis for searching the knowledge base.

Automatic download: If you do not provide content, we automatically download the document based on the given URL and extract the content. This is convenient when you do not want to fetch and extract the content yourself.

Is the URL behind a login, or does your server ask for a key? Then set the required headers and parameters on the API source in the app, under Attachment extraction, for example an Authorization header. We send them along when fetching the content of the URL, and also when fetching attachments.

Please note
In the case of "mime_type": "text/html" we expect HTML that is as complete as possible and valid.

Always wrap content in full HTML tags: <html><body> ... </body></html>

Additional fields

`meta_data`

Type: object (optional)

A flexible object for additional metadata that can enrich the behavior of the knowledge base and search results. You are completely free to add your own fields here. This metadata is used to make the systems smarter and can be used to enrich search results. The fields you send appear in the app on the source under Metadata, marked With document, and are immediately usable in filters and search strategies.

Commonly used `meta_data` fields (examples):

Although there is no prescribed structure, the following fields are frequently used and recognized by the system:

author (string) - The author or creator of the document
date (string, ISO 8601 format) - Publication or modification date of the document. This date can be used to determine how up to date sources are.
language (string, ISO 639-1 code) - The language of the document (e.g. "nl", "en")
category (string) - Categorization of the document (e.g. "product documentation", "blog post", "manual")

Using custom fields:

You are completely free to add your own metadata fields that are specific to your use case. These custom fields can be used for:

Custom functionality within systems
Making your Agent and Answer system smarter
Filtering and faceting of search results
Enriching your search results

{
  "data": {
    "external_reference": "scrapedPage:9001",
    "url": "https://www.domein-met-het-nieuwe-zoeken.nl/over-ons",
    "mime_type": "text/html",
    "title": "About us",
    "content": "<html><body><p>Hello world!</p></body></html>",
    "meta_data": {
        "date": "2024-12-12T12:00:00+00:00"
    }
  }
}

Nested objects

If you send an object inside a field, we turn it into separate fields: the names are joined with an underscore. A list of values stays a single field with multiple values. So this:

"meta_data": {
    "author": {
        "name": "Jane Smith",
        "department": "Communications"
    },
    "tags": ["parking", "permit"]
}

produces the fields author_name, author_department and tags. That is how you see them in the app, and how you use them in filters.

Endpoints

Creating a new document

POST https://{klant}.vragen.ai/api/v1/documents

Example request body:

{
    "data": {
        "type": "documents",
        "attributes": {
            "url": "https://domein-met-het-nieuwe-zoeken.nl",
            "title": "Homepage",
            "content": "<html><body><p>SWIS has everything in house to discover what your organization needs to become digitally successful. And stay that way. Discover what you really need. Starting from a successful strategy, we conceive and design a solution. More about discovering Making what is most valuable. Our agile teams know exactly how to build such a solution. In a smart, fast and smooth process. More about making. Optimizing to stay successful. That means you need to keep turning the dials of your product or service. We help you with that. More about optimizing</p></body></html>",
            "external_reference": "foo_1",
            "mime_type": "text/html",
            "meta_data": {
                "author": "SWIS"
            }
        }
    }
}

Example response

{
    "jsonapi": {
        "version": "1.0"
    },
    "links": {
        "self": "https://{klant}.vragen.ai/api/v1/documents/9001"
    },
    "data": {
        "type": "documents",
        "id": "9001",
        "attributes": {
            "external_reference": "foo_1",
            "url": "https://domein-met-het-nieuwe-zoeken.nl",
            "mime_type": "text/html",
            "title": "Homepage",
            "meta_data": {
                "author": "SWIS"
            },
            "content_cleaned": "..." 
        },
        "links": {
            "self": "https://{klant}.vragen.ai/api/v1/documents/9001"
        }
    }
}

Reading an existing document

GET https://{klant}.vragen.ai/api/v1/documents/{document_id}

Example response (similar to 'Creating a new document'):

{
    "jsonapi": {
        "version": "1.0"
    },
    "links": {
        "self": "https://{klant}.vragen.ai/api/v1/documents/9001"
    },
    "data": {
        "type": "documents",
        "id": "9001",
        "attributes": {
            "external_reference": "foobar_1",
            "url": "https://domein-met-het-nieuwe-zoeken.nl",
            "mime_type": "text/html",
            "title": "Homepage",
            "meta_data": null,
            "content_cleaned": "This is all the content of the page."
        },
        "links": {
            "self": "https://{klant}.vragen.ai/api/v1/documents/9001"
        }
    }
}

Updating an existing document

PATCH https://{klant}.vragen.ai/api/v1/documents/{document_id}

Example request body:

{
    "data": {
        "id": "9001",
        "type": "documents",
        "attributes": {
            "url": "https://domein-met-het-nieuwe-zoeken.nl",
            "external_reference": "foo_1",
            "mime_type": "text/html",
            "title": "Homepage",
            "content": "<html><body><p>SWIS has everything in house to discover what your organization needs to become digitally successful. And stay that way. Discover what you really need. Starting from a successful strategy, we conceive and design a solution. More about discovering Making what is most valuable. Our agile teams know exactly how to build such a solution. In a smart, fast and smooth process. More about making. Optimizing to stay successful. That means you need to keep turning the dials of your product or service. We help you with that. More about optimizing</p></body></html>",
            "meta_data": {
                "author": "SWIS - test"
            }
        }
    }
}

Example response:

{
    "jsonapi": {
        "version": "1.0"
    },
    "links": {
        "self": "https://{klant}.vragen.ai/api/v1/documents/9001"
    },
    "data": {
        "type": "documents",
        "id": "9001",
        "attributes": {
            "external_reference": "foo_1",
            "url": "https://domein-met-het-nieuwe-zoeken.nl",
            "mime_type": "text/html",
            "title": "Homepage",
            "meta_data": {
                "author": "SWIS - test"
            },
            "content_cleaned": "This is all the content of the page."
        },
        "links": {
            "self": "https://{klant}.vragen.ai/api/v1/documents/9001"
        }
    }
}

Reading all available documents

GET https://{klant}.vragen.ai/api/v1/documents

Example response body

{
    "meta": {
        "page": {
            "currentPage": 1,
            "from": 1,
            "lastPage": 901,
            "perPage": 10,
            "to": 10,
            "total": 9001
        }
    },
    "jsonapi": {
        "version": "1.0"
    },
    "links": {
        "first": "https://{klant}.vragen.ai/api/v1/documents?page%5Bnumber%5D=1&page%5Bsize%5D=10",
        "last": "https://{klant}.vragen.ai/api/v1/documents?page%5Bnumber%5D=634&page%5Bsize%5D=10",
        "next": "https://{klant}.vragen.ai/api/v1/documents?page%5Bnumber%5D=2&page%5Bsize%5D=10"
    },
    "data": [
        {
            "type": "documents",
            "id": "1",
            "attributes": {
                "external_reference": "scrapedPage:119145",
                "url": "https://www.domein-met-het-nieuwe-zoeken.nl/nieuws",
                "mime_type": "text/html",
                "title": "Actueel | Het Nieuwe Zoeken",
                "meta_data": {
                    "date": "2023-07-19T12:00:00+00:00"
                },
                "content_cleaned": "Today we start searching, anno 2024"
            },
            "links": {
                "self": "https://{klant}.vragen.ai/api/v1/documents/1"
            }
        },
        ...
      ]
  }

The response will contain a list of documents, if any are present. The structure of these objects is the same for each document and similar to when you request a single document.

Please note: with the following information returned in the response, you can send follow-up requests to retrieve further information

{
  ... 
  "meta": {
    "page": {
        "currentPage": 1,
        "from": 1,
        "lastPage": 901,
        "perPage": 10,
        "to": 10,
        "total": 9001
    }
  },
  "links": {
      "first": "https://{klant}.vragen.ai/api/v1/documents?page%5Bnumber%5D=1&page%5Bsize%5D=10",
      "last": "https://{klant}.vragen.ai/api/v1/documents?page%5Bnumber%5D=634&page%5Bsize%5D=10",
      "next": "https://{klant}.vragen.ai/api/v1/documents?page%5Bnumber%5D=2&page%5Bsize%5D=10"
  },
  ...
}

Deleting a document

DELETE https://{klant}.vragen.ai/api/v1/documents/{document_id}

Support

Stuck? Email our support team at service@vragen.ai.

Still have a question?

Ask it here. The answer comes from this documentation and points to the right page.

You are talking to an AI assistant from vragen.ai. Why we mention this

Connecting via the API

Documents

Before you start

Which source do your documents land in?

API documentation

General information and principles

Document schema

Basic fields

external_reference

url

mime_type

title

content

Additional fields

meta_data

Commonly used meta_data fields (examples):

Using custom fields:

Nested objects

Endpoints

Creating a new document

Reading an existing document

Updating an existing document

Reading all available documents

Deleting a document

Support

Still have a question?

`external_reference`

`url`

`mime_type`

`title`

`content`

`meta_data`

Commonly used `meta_data` fields (examples):