> ## Documentation Index > Fetch the complete documentation index at: https://docs.verbex.ai/llms.txt > Use this file to discover all available pages before exploring further. # Crawl Web Pages > Crawls the given pages for asynchronous ingestion. One call creates one document covering every URL in the list, not one document per URL. Like file upload, a 202 means queued, not indexed - poll the document status endpoint until it reports 'completed'. There is no document update endpoint. Re-crawling the same URLs creates a second document while the first remains; to refresh, delete and re-crawl. ## OpenAPI ````yaml /api-reference/openapi.json post /api/v1/knowledge-bases/{kb_id}/documents/website openapi: 3.1.0 info: title: Verbex Platform API description: API for managing AI agents, calls, phone numbers, and more. version: 1.0.0 servers: [] security: [] tags: - name: Knowledge Bases description: >- Create and manage knowledge bases, ingest documents into them, and search them semantically. **Authentication.** Every request carries a bearer token: `Authorization: Bearer `. That is the only credential a client sends. The gateway resolves your identity from it and injects the tenant headers the service reads internally, so clients never send `x-user-org-id` or `x-verbex-id` themselves. **Tenancy is enforced on every route.** A knowledge base belongs to one organization and one user. Requesting one your token does not own returns `404 Knowledge base not found`, never `403`. This is deliberate: a nonexistent ID, a malformed ID and someone else's ID are indistinguishable in the response, so the API cannot be used to probe for other tenants' data. If you get a `404` on an ID you are certain exists, suspect the token before you suspect the ID. **Ingestion is asynchronous.** File upload and website crawl both answer `202` as soon as the material is stored and the job is queued. Parsing, chunking, embedding and indexing happen afterwards in a worker. Poll the document status endpoint until it reports `completed` before expecting search to see the content. That gap is the most common source of "why isn't my data showing up". **There is no document update endpoint.** Every submission mints a new `document_id`, so re-uploading a file or re-crawling the same URLs creates a second document while the first remains. To refresh material, delete then resubmit — and note that the sequence is not atomic. paths: /api/v1/knowledge-bases/{kb_id}/documents/website: post: tags: - Knowledge Bases summary: Crawl Web Pages description: >- Crawls the given pages for asynchronous ingestion. One call creates one document covering every URL in the list, not one document per URL. Like file upload, a 202 means queued, not indexed - poll the document status endpoint until it reports 'completed'. There is no document update endpoint. Re-crawling the same URLs creates a second document while the first remains; to refresh, delete and re-crawl. operationId: >- add_website_document_api_v1_knowledge_bases__kb_id__documents_website_post parameters: - name: kb_id in: path required: true schema: type: string title: Kb Id description: Knowledge Base ID. requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/CreateWebsiteDocumentRequest' responses: '202': description: Crawl accepted and queued. Not yet searchable. content: application/json: schema: $ref: '#/components/schemas/DocumentSubmissionResponse' '401': description: Missing or invalid bearer token. content: application/json: schema: $ref: '#/components/schemas/KnowledgeBaseErrorResponse' '404': description: >- Knowledge base not found, or not owned by your tenant. The two cases are deliberately indistinguishable. content: application/json: schema: $ref: '#/components/schemas/KnowledgeBaseErrorResponse' '409': description: Integrity conflict, for example concurrent writes to the same key. content: application/json: schema: $ref: '#/components/schemas/KnowledgeBaseErrorResponse' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' '503': description: >- Service unavailable. A database failure, not a problem with your payload. content: application/json: schema: $ref: '#/components/schemas/KnowledgeBaseErrorResponse' components: schemas: CreateWebsiteDocumentRequest: properties: page_urls: items: type: string type: array title: Page Urls description: >- Pages to crawl. One call creates one document covering every URL in this list, not one document per URL. type: object required: - page_urls title: CreateWebsiteDocumentRequest description: Request body for crawling web pages into a knowledge base. example: page_urls: - https://example.com/faq - https://example.com/pricing DocumentSubmissionResponse: properties: document_id: type: string title: Document Id description: >- Server-minted identifier for the submitted document. Keep it - you need it to poll the status endpoint. status: $ref: '#/components/schemas/DocumentProcessingStatus' description: Always 'processing' at submission time. message: anyOf: - type: string - type: 'null' title: Message description: Human-readable detail, for example 'File processing started'. type: object required: - document_id - status title: DocumentSubmissionResponse description: Acknowledgement that a document was accepted for asynchronous ingestion. KnowledgeBaseErrorResponse: properties: error: type: string title: Error description: A short error code identifying the type of error that occurred. message: type: string title: Message description: >- A detailed human-readable message explaining the error and possible solutions. type: object required: - error - message title: KnowledgeBaseErrorResponse HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError DocumentProcessingStatus: type: string enum: - processing - completed - failed title: DocumentProcessingStatus description: >- Ingestion state of a document. Only 'completed' means the material is searchable. ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type type: object required: - loc - msg - type title: ValidationError ````