Go to the Knowledge Base Section

Create a new group to organize your documentation.

Select Docusaurus as your documentation source if your documentation system is based on Docusaurus

Select Web to use web-based documentation. This works for most of the use cases as it crawls the web page and all the links it contains inside

Give the group a name so that it gets easy to manage and maintain it

Enter the URL of your web-based documentation.

Hit Create button to create it with the above given details

The new group is successfully created now!

Navigate to the Knowledge Base section.

You can confirm that the new group is created and listed under the Knowledge page as a group

We are all set to fetch the content and add them as documentation for our chatbot. Hit Refresh button

You can check the progress of the fetch process. It shows the number of pages it fetches, the current status, and you can stop it in between if you want.

Navigate to the group to find out more details

Go to Knowledge Items tab to view the pages it has fetched and the contents.

You can see the pages it has scraped. It shows the URL and Title of the page for a better understanding

You can click any item to view the contents of the page it scraped.

You can see the contents it fetched. It automatically converts the page contents to Markdown so that AI understands it better.

Remember, if any unnecessary content is fetched, it is important to remove them as it degrades the chatbot performance. This is covered in other guide.