Develop Connectors with the Fluid Topics API - Fluid Topics

Introduction to developing Fluid Topics connectors

The Fluid Topics Python library allows users to develop Fluid Topics connectors.

When developing a Fluid Topics connector, it is necessary to install the Fluid Topics Python library first. The fluidtopics Python package is available in the official Python Package Index.

The following line shows how to install the Fluid Topics Python package:

pip install fluidtopics

Create a connector

Creating a connector requires importing the appropriate objects from the fluidtopics.connector library:

from fluidtopics.connector import RemoteClient, LoginAuthentication, StructuredDocument, Topic

Most of the Fluid Topics Python API objects (StructuredDocument, UnstructuredDocument, ExternalDocument, Metadata...) are immutable. Once created, it is not possible to modify them. To update an object, it is necessary to use the .update() method.

Configure authentication

The LoginAuthentication object allows developers to set up the user credentials that are sent to the RemoteClient object.

The RemoteClient object allows the connector to communicate with Fluid Topics.

from fluidtopics.connector import LoginAuthentication, RemoteClient

# First, create a RemoteClient
auth = LoginAuthentication('khub_admin_user@domain.com', 'user_password')
client = RemoteClient('https://my-fluidtopics.com', auth, 'external_source_id')

The Fluid Topics Python library requires an authentication using an account and password. It is not possible to use an API key.

Create a source

To publish documents, it is necessary to create a Fluid Topics external source, either manually or programmatically:

from fluidtopics.connector import LoginAuthentication, RemoteClient

# Firstly, create a RemoteClient
auth = LoginAuthentication("khub_admin_user@domain.com", "user_password")
client = RemoteClient("https://my-fluidtopics.com", auth, "external_source_id")

# Secondly, create the source
client.create_source(
    name="External source",
    description="External source for the connector")

The name and description attributes are optional.

Build publications

This section contains information about how to build unstructured, structured, and external documents.

Build an unstructured document

The following lines show how to build an unstructured document:

from fluidtopics.connector import UnstructuredDocument

ud = UnstructuredDocument.from_uri(
    document_id="my-UD",
    title="I am a UD",
    locale="en-GB",
    description="Description of a UD",
    pretty_url="my-pretty-UD-url",
    metadata=[meta_simple_38],
    uri="/path/to/a/ud.pdf"
)

The uri parameter can be a file path or a URL.

Build a structured document

Structured documents are composed of topics which are arranged in a table of contents.

Create a topic

A topic defines a table of contents level and is typically composed of the following elements:

An ID
A title
An HTML body
One or more child topics (subsections in the table of contents)

The HTML body must contain only a fragment of the HTML file and not the file in its entirety.

The following lines show how to build topics:

from fluidtopics.connector import Topic

sub_topic = Topic.create(
    topic_id="child-topic-id",
    title="Child topic title",
    body="<div><p>Body of the <strong>child</strong> topic</p></div>"
)

topic_A = Topic.create(
    topic_id="topic-a-id",
    title="Topic A title",
    body="<div><p>Body of the topic <strong>A</strong></p></div>",
    children=[sub_topic]
)

topic_B = Topic.create(
    topic_id="topic-b-id",
    title="Topic B title",
    body="<div><p>Body of the topic <strong>B</strong></p></div>",
    children=[]
)

Create a resource

The following lines shows how to build a resource:

from fluidtopics.connector import Resource

resource = Resource.from_uri(
    resource_id="my-resource-img",
    uri="/path/to/an/image.jpg"
)

topic_A = Topic.create(
    topic_id="topic-a-id",
    title="Topic A title",
    body='<p>Body of the topic <strong>A</strong></p><p><img src="my-resource-img"/></p>'
)

document = StructuredDocument.create(
    document_id='docId',
    title='Title',
    locale='en-US',
    toc=[topic_A],
    resources=[resource],
    editorial_type=EditorialType.BOOK
)

Create a structured document

The following lines show how to build structured documents:

from fluidtopics.connector import EditorialType, StructuredDocument

# Book
book_A = StructuredDocument.create(
    document_id="my-structured-document-book-a",
    title="My Structured Document - BookA",
    locale="en-US",
    pretty_url="my-pretty-structured-document-book-a",
    toc=[topic_A, topic_B],
    editorial_type=EditorialType.BOOK
)

# Article
article = StructuredDocument.create(
    document_id="my-structured-document-article",
    title="My Structured Document - Article",
    locale="en-US",
    pretty_url="my-pretty-structured-document-article",
    toc=[topic_B],
    editorial_type=EditorialType.ARTICLE
)

Add attachments

An attachment is a file or a URL attached to a structured document. Examples of attachments include JavaScript files, PDFs, image files, video files, and ZIP archives.

The following lines show how to add locally and externally sourced attachments:

from fluidtopics.connector import StructuredDocument, Attachment

# Attachment creation
attachment = Attachment.from_uri(
    attachment_id="my-attachment",
    uri="/local/path/to/my/attachment.zip",
    title="My Attachment",
    filename="name-of-my-file.zip"
)

# External attachment creation
external_attachment = Attachment(
    attachment_id="my-external-attachment",
    title="FT Global Presence",
    mime_type="video/mpeg",
    url="https://www.youtube.com/watch?v=hsXwJb-agsQ",
    filename="name-of-my-file.mp4"
)

book = StructuredDocument.create(
    document_id="my-structured-document",
    title="My Structured Document",
    locale="en-US",
    toc=[topic_A, topic_B],
    attachments=[attachment, external_attachment]
)

Build an external document

The following lines show how to build an external document:

from fluidtopics.connector import ExternalDocument

external_document = ExternalDocument.create(
    document_id="my-external-document",
    title="my External Document",
    url="https://www.fluidtopics.com"
)

Finalize publications

Before publishing content, it is possible to enrich it with metadata and configure its appearance in the portal.

Metadata

Metadata are a combination of keys and values which users can add to publications (structured, unstructured, or external) and topics. Metadata improve the user experience throughout the portal.

Fluid Topics supports the following types of metadata:

String metadata
Hierarchical string metadata
Built-in metadata

Add metadata

String metadata

String metadata associates a string value with a key. String metadata can be flat or hierarchical. String metadata can also include multiple values, for example, all the authors of a topic:

from fluidtopics.connector import Topic, Metadata

string_meta = Metadata("version", "3.7")
multivalued_string_meta = Metadata("author", ["John Doe", "Jane Smith"])

topic_A = Topic.create(
    topic_id="topic-a-id",
    title="Topic A title",
    body="<p>Body of the topic <strong>A</strong></p>",
    children=[sub_topic],
    metadata=[string_meta, multivalued_string_meta]
)

Hierarchical string metadata

Hierarchical string metadata associates a tree of string values with a key. For instance, if the metadata is configured as a search facet, Fluid Topics allows users to search for all content matching one of the nodes of the hierarchy.

The following lines show an example of hierarchical string metadata:

from fluidtopics.connector import StructuredDocument, Metadata

hierarchical_string_meta = Metadata(
    "Antidot",
    [
        ["Solutions", "Fluid Topics"],
        ["Solutions", "AFS@Store"],
        ["Technology", "Content Delivery"],
        ["Technology", "Taruqa Search"],
        ["Technology", "Content Classifier"]
    ]
)

book_A = StructuredDocument.create(
    document_id="my-structured-document-book-a",
    title="My Structured Document - BookA",
    locale="en-US",
    pretty_url="my-pretty-structured-document-book-a",
    toc=[topicA, topicB],
    editorial_type=EditorialType.BOOK,
    metadata=[hierarchical_string_meta]
)

Built-in metadata

The following lines show how to create built-in metadata:

from fluidtopics.connector import EditorialType, StructuredDocument, Metadata

# Create built-in metadata
title = Metadata.title("Document Title")
locale = Metadata.locale("en-US")
base_id = Metadata.base_id("my-base-id)
pretty_url = Metadata.pretty_url("my-pretty-url")
editorial_type = Metadata.editorial_type(EditorialType.BOOK)
last_edition_date = Metadata.last_edition('2020-06-18')

# Attach built-in metadata to a publication
book_B = StructuredDocument.create(
    document_id="my-structured-document"
    toc=[topicA, topicB],
    metadata=[title, locale, base_id, pretty_url, editorial_type, last_edition_date],
)

# Or use dedicated method attributes to attach a built-in metadata
article = StructuredDocument.create(
    document_id="my-structured-document-article", 
    title="Article title",
    locale="en-US",
    base_id="my-article-base-id" 
    pretty_url="my-pretty-article",
    toc=[topic_B],
    editorial_type=EditorialType.ARTICLE,
    last_edition="2020-05-15"
)

In the example above, title = Metadata.title('Document Title') corresponds to the built-in metadata ft:title.

Update metadata journal

Whenever metadata is updated, it is necessary to rebuild the metadata journal to display the updated metadata.

The following lines show how to rebuild a metadata journal:

original_metadata = Metadata("metadataKey", "originalValue")
updated_value = [original_metadata.update()
                 for value in original_metadata.value]
normalized_value = original_metadata.update(
    value=updated_value,
    producer="Normalizer",
    comment="Update metadata value")

Example:

The following example shows how to capitalize all the letters of the names of content authors that were originally defined using lowercase letters:

original_authors = Metadata("author", ["jane", "John"])
capitalized_metadata = [author.capitalize() for author in original_authors.value]
normalized_authors = original_authors.update(value=capitalized_values, producer="Normalizer", comment="Cap

Inherit metadata

By default, when using the Fluid Topics Python API to create metadata at the book level, the book's topics do not inherit the metadata.

It is possible to override the API's default behavior so that a book's topics inherit metadata created at the book level, as well as a topic's child topics inherit the metadata created at the topic level. For this purpose, the API provides a MetadataInheritor object.

Built-in metadata cannot be inherited.

As an example, when a topic already has a metadata, the MetadataInheritor object adds the metadata of the book:

Before inheritance: Book(meta = value1) > Topic(meta = value2)
After inheritance: Book(meta = value1) > Topic(meta = value1, value2)

The following lines show how to enable metadata inheritance for a given book:

from fluidtopics.connector import Topic, EditorialType, StructuredDocument, Metadata, MetadataInheritor


meta_simple_37 = Metadata("version", "3.7")
meta_simple_rhel = Metadata("Operating System", "RHEL")

topic_B = Topic.create(
    topic_id="topic-b-id",
    title="Topic B title",
    body="<p>Body of the topic <strong>B</strong></p>"
)

book_B = StructuredDocument.create(
    document_id="my-structured-document-book-b",
    title="My Structured Document - BookB",
    locale="en-US",
    pretty_url="my-pretty-structured-document-book-b",
    toc=[topicA, topicB],
    editorial_type=EditorialType.BOOK,
    metadata=[meta_simple_37, meta_simple_rhel]
)

book_B_final = MetadataInheritor().inherit_metadata(book_B)

It is also possible to select the metadata that will not be inherited:

book_B_final = MetadataInheritor(
    ['version', 'release_date']).inherit_metadata(book_B)

Generate clusters

Technical documentation often contains chunks of information that authors reuse, for example, across product versions. As a result, different documents can be nearly identical. Rather than list similar documents one after the other, Fluid Topics aims to improve the search experience by providing a way to assemble them within a cluster. The cluster is then displayed as a single search result.

By default, Fluid Topics identifies similar documents by scanning the base_id of each document. Documents or topics with the same base_id are clustered together.

The following example shows how it is possible to create a cluster containing two documents about the same product. A selector in the Search page allows end-users to choose which document seems the most relevant.

from fluidtopics.connector import Topic, StructuredDocument, Metadata

version_1 = Metadata("product_version", ["1.0"])
version_2 = Metadata("product_version", ["2.0"])
base_id_intro = "product-X-intro"
base_id_product_x = "product-X"

# Book about "Product X" in version 1.0

product_X_intro_v1 = Topic.create(
    topic_id="product-X-intro-version-1.0",
    title="My Topic Title",
    body="Introduction of Product X ... version 1.0 ...",
    metadata=[version_1],
    base_id=base_id_intro
)

product_X_v1 = StructuredDocument.create(
    document_id="product-X-1.0",
    title="Product X - version 1.0",
    locale="en-US",
    toc=[product_X_intro_v1],
    metadata=[version_1],
    base_id=base_id_product_x
)

# Book about "Product X" in version 2.0
# Almost the same as version 1.0, with some new features.
# Most content is the same: only the version number changes

product_X_intro_v2 = Topic.create(
    topic_id="product-X-intro-version-2.0",
    title="My Topic Title 2",
    body="Introduction of Product X ... version 2.0 ...",
    metadata=[version_2],
    base_id=base_id_intro
)

product_X_v2 = StructuredDocument.create(
    document_id="product-X-2.0",
    title="Product X - version 2.0",
    locale="en-US",
    toc=[product_X_intro_v2],
    metadata=[version_2],
    base_id=base_id_product_x
)

Link content

Once documents have a unique and stable ID and once the app potentially clusters them, then it is possible to create one of the following types of links.

Link to another topic in the same document

The following examples show two ways of generating a link to another topic in the same document:

Use <a href="contentId">mylink</a> where contentId is the content identifier
Use <a href="topic_baseId" data-ft-link-type="baseId">my link</a> where topic_baseId is the topic's base_id

When no base_id is provided for a document or topic, Fluid Topics uses its document_id or content_id as a fallback value.

Link to another topic in a different document (cross-document links)

The following example show how to generate a link to another topic contained in a different document:

Use <a href="topic_baseId" data-ft-link-type="baseId">my link</a>.

Link to a resource

The following example shows how to include a resource in the document (image, zip file, and so on):

Put the resource ID in the HTML source field as follows:
- <img src="resourceId">
- <object data="resourceId">
- ...

The following line shows how two ways of including a link to a resource in the Viewer page (PDF, image, and so on):

Use <a href="id">my link</a>
<img src="imageId"/> directly for an image.

Link to external content

The following line show how to generate a link to an external site:

Use <a href="https://mysite.com/mypage.html">my link</a>.

All absolute links (beginning with https://, http://, ftp://...) are considered to be external links.

The following lines show an example of links used in Topics objects:

from fluidtopics.connector import Resource, Topic, EditorialType, StructuredDocument, Metadata

sub_topic = Topic.create(
    content_id="child-topic-id",
    title="Child topic title",
    body="<p>Body of the <strong>child</strong> topic</p>"
)

topic_A = Topic.create(
    content_id="topic-a-id",
    title="Topic A title",
    body="<p>Body of the topic <strong>A</strong></p><p><img src='my-resource'/></p>",
    children=[sub_topic]
)

topic_B = Topic.create(
    content_id="topic-b-id",
    title="Topic B title",
    body="<p>Body of the topic <strong>B</strong></p><p><a href='topic-a-id'>link to another topic in the <strong>same</strong> publication</a></p>",
    children=[]
)

topic_C = Topic.create(
    content_id="topic-c-id",
    title="Topic C title",
    body="<p>Body of the topic <strong>C</strong></p><p><a href='topic-a-id' data-ft-link-type='baseId'>link to another topic in <strong>another</strong> publication</a></p>",
    children=[]
)

book_A = StructuredDocument.create(
    document_id="my-structured-document-book-a",
    title="My Structured Document - BookA",
    locale="en-US",
    pretty_url="my-pretty-structured-document-book-a",
    toc=[topicA, topicB],
    editorial_type=EditorialType.BOOK
)

book_B = StructuredDocument.create(
    document_id="my-structured-document-book-b",
    title="My Structured Document - BookB",
    locale="en-US",
    pretty_url="my-pretty-structured-document-book-b",
    toc=[topic_C],
    editorial_type=EditorialType.BOOK
)

Publish content

The RemoteClient() object of the Fluid Topics Python API provides a publish() method allowing users to send documents to the tenant configured when the client was created.

The following example shows how to publish content:

# Publish
client.publish(doc1, doc2, doc3, publish_name="Uploaded from FT Python API")

publish_name is the optional name of the publish job in the Publishing administrative interface.

Example

The following lines show how to publish documents to a Fluid Topics tenant:

from fluidtopics.connector import EditorialType, LoginAuthentication, Metadata, RemoteClient, StructuredDocument, Topic, UnstructuredDocument

# Client creation
authentication = LoginAuthentication("root@fluidtopics.com", "change_it")
client = RemoteClient(
    "http://localhost:8080/myPortal",
    authentication,
    "external")

# Metadata creation
meta1 = Metadata("version", "4.2")
meta2 = Metadata("author", "John Doe")

# Topics creation
topic_A = Topic.create(topic_id="topic-id-a", title="Topic A")
topic_B = Topic.create(topic_id="topic-id-b", title="Topic B")

# Unstructured Document creation
ud = UnstructuredDocument.from_uri(
    document_id="my-UD",
    title="I am a UD",
    locale="en-GB",
    description="Description of a UD",
    pretty_url="my-pretty-UD",
    metadata=[meta1, meta2],
    uri="/path/to/the/file/ud.pdf"
)

# Structured Document creation
book = StructuredDocument.create(
    document_id="my-structured-document",
    title="My Structured Document",
    locale="en-US",
    toc=[topic_A, topic_B],
    editorial_type=EditorialType.BOOK
)

# Publish
client.publish(ud, book, publish_name="Uploaded from FT Python API")

Test a connector

Before packaging a connector, it is highly recommended to test it. The Fluid Topics Python API provides the FakeClient object for testing purposes.

The following lines show a basic connector code and an example of a test file for it.

hw.py (connector code):

from fluidtopics.connector import StructuredDocument, Topic, Metadata, Client


class HelloWorldConnector:
    def __init__(self, client: Client):
        self.client = client

    def run(self):
        hello = StructuredDocument.create(
            document_id="hello",
            title="Hello",
            toc=[Topic.create(topic_id="hello-topic", title="Hello topic")],
            metadata=[Metadata("tutorial", "HelloWorld")]
        )

        world = StructuredDocument.create(
            document_id="world",
            title="World",
            toc=[]
        )

        self.client.publish(hello, world)

test_helloworld.py (testing the hw.py connector):

from fluidtopics.connector import StructuredDocument, Metadata, FakeClient
from hw import HelloWorldConnector


def test_hello_title():
    client = FakeClient()
    connector = HelloWorldConnector(client)

    connector.run()

    hello = client.publication("hello")  # assert that document is published
    assert hello.title == "Hello"


def test_hello_metadata():
    client = FakeClient()
    connector = HelloWorldConnector(client)

    connector.run()

    hello = client.publication("hello")
    # should fail
    assert Metadata("invalid", "metadata") in hello.metadata.values()


def test_hello_toc_length():
    client = FakeClient()
    connector = HelloWorldConnector(client)

    connector.run()

    hello = client.publication("hello")
    assert isinstance(hello, StructuredDocument)
    assert len(hello.toc) == 1

Inform users

Use the logging module to inform users through the job report.

Example:

import logging
logger = logging.getLogger(__name__)
logger.info(f'"{NUMBER_TOPICS}" is used to create TOC.')
logger.warning(f"No document is present in the archive: a root {README_PATH} file or a {TOC_PATH} is mandatory to upload a Markdown document.")

logger.info creates the following kind of message:

A log entry interface showing a log with two columns: 'Date' and 'Message'. The message is highlighted in blue, stating '"README.md" is used to create TOC.' An information icon precedes the message.

logger.warning creates the following kind of message:

A log entry interface showing a log with two columns: 'Date' and 'Message'. The message is highlighted in orange, stating 'No document is present in the archive: a root README.md file or a toc.yml is mandatory to upload a markdown document.' A warning icon precedes the message.

When raising an exception, Fluid Topics creates the following kind of message:

A log entry interface showing a log with two columns: 'Date' and 'Message'. The message is highlighted in red, stating 'Unable to find topics ../docs/example.md' An error icon precedes the message.

Package a connector

It is necessary to package the connector before distributing it. This operation consists in packaging a Python project.

Get in touch with a Fluid Topics representative to deploy a custom connector.

Basic connector example

The following lines provide a basic example to copy and paste in a Python project.

Modify the login credentials, the source ID, and the tenant URL to get a new connector up and running.

from fluidtopics.connector import EditorialType, ExternalDocument, LoginAuthentication, Metadata, RemoteClient, StructuredDocument, Topic


def main():
    authentication = LoginAuthentication("root@fluidtopics.com", "change_it")
    client = RemoteClient(
        "http://localhost:8080/myPortal",
        authentication,
        "external")
    client.create_source(
        name="External",
        description="External source for my connector")

    topic_a = Topic.create(
        topic_id="topic-id",
        title="I am a topic",
        body="<p>Hello World!</p>"
    )

    topic_b = Topic.create(
        topic_id="topic-id-2",
        title="I am another topic",
        body="<p>Hi Earth!</p>"
    )

    author = Metadata("author", "John Doe")
    version = Metadata("version", "4.2")
    category = Metadata("category", "Demo")

    doc = StructuredDocument.create(
        document_id="document-id",
        title="Hello World Book",
        toc=[topic_a, topic_b],
        locale="en-US",
        metadata=[author, version, category],
        editorial_type=EditorialType.BOOK
    )

    external_doc = ExternalDocument.create(
        document_id="external-document",
        title="Hello Fluid Topics!",
        url="https://www.fluidtopics.com",
        metadata=[author]
    )

    client.publish(
        doc,
        external_doc,
        publish_name="Uploaded from Fluid Topics Python API")


if __name__ == "__main__":
    main()

This code generates and publishes a structured document with two topics and an external document.