> For the complete documentation index, see [llms.txt](https://docs.clearfeed.ai/clearfeed-help-center/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.clearfeed.ai/clearfeed-help-center/clearfeed-ai/knowledge-sources/indexing-knowledge-sources/html.md).

# HTML

In an HTML knowledge source, you can enter the URL of any public website to be indexed and used for answer generation.

* You can specify conditions to consider only specific pages/sub-pages under a Root URL
* ClearFeed can automatically refresh the data from the source every 24 hours if you keep Auto-Reindex turned on.

{% hint style="info" %}
**Common Use Cases:** Product Documentation, Blogs, Case Studies, Public Websites, etc...
{% endhint %}

#### Add HTML Knowledge Source

* Go to **KBs** from the left navigation bar and click the **"Add Knowledge Source"** dropdown in the top-right corner. Select **HTML**.
* Fill out the side drawer form:
  * **Name**: Enter a title for the Knowledge Source.
  * **Tags**: Add tags to categorize the Knowledge Source.
  * **Root URL**: Provide the main URL ClearFeed will index from. Only URLs branching from this root will be indexed. For example, if the root is `https://docs.clearfeed.ai/clearfeed-help-center/product-features/ticketing`, a URL like `https://docs.clearfeed.ai/clearfeed-help-center/product-features/collections` will not be indexed.
  * **URL Conditions**: Set rules for URLs to include/exclude using options like "Starts with," "Does not start with," "Contains," or "Does not contain."
  * **Auto Reindex Regularly**: Enable this to reindex the Knowledge Source every 24 hours.
* Save your entry. The Knowledge Source will either start indexing immediately (**In Progress** status) or shortly after (**Initiated** status). Get more details on Managing Knowledge Sources here.
* **Reindexing Public Docs -** To reindex public documents, **manually reindex** the knowledge source by clicking the "Reindex" button next to it.

{% hint style="info" %}
**Optimize Indexing of Knowledge Sources with Sitemaps**

ClearFeed will check each root URL for sitemaps, as using sitemaps is faster than a full website crawl. If a sitemap is found, we'll fetch all pages under the specified root URL, excluding any URLs you've chosen to exclude, ensuring all relevant pages are considered.
{% endhint %}

#### Using AI Agents

Once you've configured the knowledge sources, learn how to build and use AI Agents

{% content-ref url="/pages/BG6lBe8zAALu5HQvWnQQ" %}
[Build AI Agents](/clearfeed-help-center/clearfeed-ai/ai-agents/build-ai-agents.md)
{% endcontent-ref %}

{% content-ref url="/pages/KfSu1GlJW2NmCvxVTv5c" %}
[Use AI Agents](/clearfeed-help-center/clearfeed-ai/ai-agents/use-ai-agents.md)
{% endcontent-ref %}

#### **Manage Knowledge Sources**

Once the status of the knowledge source has been indexed, learn more on how to manage the knowledge source below.

{% content-ref url="/pages/4mEyGIe8R45x3rjcY7ug" %}
[Manage](/clearfeed-help-center/clearfeed-ai/knowledge-sources/managing-knowledge-sources.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.clearfeed.ai/clearfeed-help-center/clearfeed-ai/knowledge-sources/indexing-knowledge-sources/html.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.