# HTML

In an HTML knowledge source, you can enter the URL of any public website to be indexed and used for answer generation.

* You can specify conditions to consider only specific pages/sub-pages under a Root URL
* ClearFeed can automatically refresh the data from the source every 24 hours if you keep Auto-Reindex turned on.

{% hint style="info" %}
**Common Use Cases:** Product Documentation, Blogs, Case Studies, Public Websites, etc...
{% endhint %}

#### Add HTML Knowledge Source

* Go to **KBs** from the left navigation bar and click the **"Add Knowledge Source"** dropdown in the top-right corner. Select **HTML**.
* Fill out the side drawer form:
  * **Name**: Enter a title for the Knowledge Source.
  * **Tags**: Add tags to categorize the Knowledge Source.
  * **Root URL**: Provide the main URL ClearFeed will index from. Only URLs branching from this root will be indexed. For example, if the root is `https://docs.clearfeed.ai/clearfeed-help-center/product-features/ticketing`, a URL like `https://docs.clearfeed.ai/clearfeed-help-center/product-features/collections` will not be indexed.
  * **URL Conditions**: Set rules for URLs to include/exclude using options like "Starts with," "Does not start with," "Contains," or "Does not contain."
  * **Auto Reindex Regularly**: Enable this to reindex the Knowledge Source every 24 hours.
* Save your entry. The Knowledge Source will either start indexing immediately (**In Progress** status) or shortly after (**Initiated** status). Get more details on Managing Knowledge Sources here.
* **Reindexing Public Docs -** To reindex public documents, **manually reindex** the knowledge source by clicking the "Reindex" button next to it.

{% hint style="info" %}
**Optimize Indexing of Knowledge Sources with Sitemaps**

ClearFeed will check each root URL for sitemaps, as using sitemaps is faster than a full website crawl. If a sitemap is found, we'll fetch all pages under the specified root URL, excluding any URLs you've chosen to exclude, ensuring all relevant pages are considered.
{% endhint %}

#### Using AI Agents

Once you've configured the knowledge sources, learn how to build and use AI Agents

{% content-ref url="../../ai-agents/build-ai-agents" %}
[build-ai-agents](https://docs.clearfeed.ai/clearfeed-help-center/clearfeed-ai/ai-agents/build-ai-agents)
{% endcontent-ref %}

{% content-ref url="../../ai-agents/use-ai-agents" %}
[use-ai-agents](https://docs.clearfeed.ai/clearfeed-help-center/clearfeed-ai/ai-agents/use-ai-agents)
{% endcontent-ref %}

#### **Manage Knowledge Sources**

Once the status of the knowledge source has been indexed, learn more on how to manage the knowledge source below.

{% content-ref url="../managing-knowledge-sources" %}
[managing-knowledge-sources](https://docs.clearfeed.ai/clearfeed-help-center/clearfeed-ai/knowledge-sources/managing-knowledge-sources)
{% endcontent-ref %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.clearfeed.ai/clearfeed-help-center/clearfeed-ai/knowledge-sources/indexing-knowledge-sources/html.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.