Public Knowledge Sources

Learn how to add and index your public Knowledge Sources

Public Knowledge Sources refer to platforms that are publicly accessible without requiring any specific permissions.

There are two types of Public Docs:

• HTML

• Site Search.

In an HTML knowledge source, you can enter the URL of any public website to be indexed and used for answer generation.

• You can specify conditions to consider only specific pages/sub-pages under a Root URL

• ClearFeed can automatically refresh the data from the source every 24 hours if you keep Auto-Reindex turned on.

Common Use Cases: Product Documentation, Blogs, Case Studies, Public Websites, etc...

Add an HTML Knowledge Source:

• Go to KBs from the left navigation bar and click the "Add Knowledge Source" dropdown in the top-right corner. Select HTML.

• Fill out the side drawer form:

  • Name: Enter a title for the Knowledge Source.

  • Tags: Add tags to categorize the Knowledge Source.

  • Root URL: Provide the main URL ClearFeed will index from. Only URLs branching from this root will be indexed. For example, if the root is https://docs.clearfeed.ai/clearfeed-help-center/product-features/ticketing, a URL like https://docs.clearfeed.ai/clearfeed-help-center/product-features/collections will not be indexed.

  • URL Conditions: Set rules for URLs to include/exclude using options like "Starts with," "Does not start with," "Contains," or "Does not contain."

  • Auto Reindex Regularly: Enable this to reindex the Knowledge Source every 24 hours.

• Save your entry. The Knowledge Source will either start indexing immediately (Indexing status) or shortly after (Initiated status). Get more details on Managing Knowledge Sources here.

• Reindexing Public Docs - To reindex public documents, manually reindex the knowledge source by clicking the "Reindex" button next to it.

Optimize Indexing of Knowledge Sources with Sitemaps

ClearFeed will check each root URL for sitemaps, as using sitemaps is faster than a full website crawl. If a sitemap is found, we'll fetch all pages under the specified root URL, excluding any URLs you've chosen to exclude, ensuring all relevant pages are considered.

Site Search

Use Site Search to create a knowledge source that looks at the internet to search for answers. When it is enabled, ClearFeed will search the web for answers. You can limit the search to a specific set of pages.

Search accuracy and latency might be lower since we do not store any data on our servers for the Site Search knowledge source

Add a Site Search Knowledge Source

• Go to KBs from the left navigation bar and click the "Add Knowledge Source" dropdown in the top-right corner. Select Site Search.

• Fill out the side drawer form:

  • Name: Enter a title for the Knowledge Source.

  • Tags: Add tags to categorize the Knowledge Source.

  • Limit Search to: Enter a set of URLs. We will only run a search among the specified URLs to look for answers.

If you don't specify any pages, ClearFeed will search the web for answers by default.

Test your Knowledge Source

Once the status of the knowledge source has been indexed, test the GPT-powered responses generated from it. Refer to the document below for more information.

Managing Knowledge Sources

3: Enable 'Answers' in the Collection Settings

Navigate to the collection settings where you want to enable GPT-Powered Answers and enable 'Answers'. Refer to the document below for more information.

Virtual Agent & Agent Assistant