Introduction

This course is aimed at frontend developers and takes you through the tasks that you will perform when working on existing Funnelback collections.

Each section contains:

  • A summary: A brief overview of what you will accomplish and learn throughout the exercise.

  • Exercise requirements: A list of requirements, such as files, that are needed to complete the exercise.

  • Detailed step-by-step instructions: detailed step-by-step instructions to guide you through completing the exercise.

  • Some extended exercises are also provided. These exercises can be attempted if the standard exercises are completed early, or as some review exercises that can be attempted in your own time.

Special tool tips or hints will appear throughout the exercises as well, which will provide extra knowledge or tips for completing the exercise. They will look like:

This box will is used to provide links to further reading available on the current topic.
This box is used to provide tips or hints related to the current topic.
This box is used to provide important advice relating to the current topic. This includes advice to be aware of to avoid common errors.
This box contains advice specific to the Linux version of Funnelback.
This box contains advice specific to the Windows version of Funnelback.

What this workshop will cover:

  • Collections / profiles / frontend services

  • Funnelback administration interface and marketing dashboard

  • Templating

  • Service creation and management

  • Best bets, synonyms and curator rules

  • Auto-completion

  • Related searches (contextual navigation)

  • Display options

  • Metadata

  • Search result filters (faceted navigation)

  • Remote includes

Prerequisites to completing the course:

  • HTML, JavaScript and CSS familiarity.

What you will need before you begin:

  • Access to the Funnelback training environment for Funnelback that has been setup for this course.

  • Internet access.

There are a number of concepts that need to be understood before you can start managing your search.

The diagram below shows users interacting with a Funnelback search.

managing search 01

Interaction between a user and Funnelback occurs via a frontend service that is configured within Funnelback. A frontend service provides the user interface and also defines the display and ranking options that apply when a search is run.

This service extends a search profile, which provides a view or scope of a collection.

A collection provides a way of searching across one or more search indexes.

An index is built for each data source or repository that you wish to provide search across.

Profiles provide access to a sub-set of the items within the search indexes.

1.1. Frontend services and profiles

A frontend service within Funnelback directly relates to a service that is provided to end users when they perform a search query.

A frontend service includes independent:

  • search templates

  • ranking and display settings

  • synonyms

  • interface functionality: faceted navigation, best bets, curator rule sets

  • knowledge graphs

  • search analytics

For example, a search conducted on a university website might include results from a number of different content sources (websites, social media, and course experts). Separate searches can then be provided for different contexts within the university website for example, a frontend service that only searches for courses (i.e. a course finder) or a search that looks at the whole university.

A service would exist in Funnelback corresponding to each of these searches. The services would contain their own usage information and configuration and would be treated as independent searches, even though they might share underlying content and indexes.

A search profile is essentially the same as a service, except it is not exposed to search administrators via the marketing dashboard.

Profiles are usually used for system-based search activities such as using Funnelback’s search index as a database for dynamic web page content.

System-based search activities don’t usually require custom templating or search analytics.

1.2. Collections

A search collection in Funnelback is similar in concept to a collection that might be housed within a library or museum.

A collection generally contains a set of items that are related in some way. In a library this could be all of the non-fiction books, or all objects relating to Charles Darwin. In Funnelback a collection is usually determined by the type of repository - e.g. a website or set of websites, a database or a fileshare repository.

Each collection in Funnelback is updated and indexed separately and can also be queried separately.

Collections in Funnelback generally include configuration to gather content from a single data source and produces a single search index that corresponds to the data source that needs to be searched.

However, there is a special collection type (a meta collection) that includes other collections - and is used to aggregate search indexes of child collections.

One or more profiles (and frontend services) can be defined within a collection to scope down the index and also provides search interface functionality, ranking and analytics.

2. The administration interface

The Funnelback administration interface is a web-based interface that provides management and administration services for a Funnelback installation.

Access to the administration interface requires a valid user login. For the training exercises the administration interface can be accessed from the following URL.

The URL of the administration interface for your organisation will vary and will usually be https://FUNNELBACK-SERVER:8443/search/admin/ where FUNNELBACK-SERVER is replaced with the host name of your Funnelback server. It is worth noting that the administration interface uses a non-standard port (port 8443) by default because the standard HTTPS port is used for public (non-admin) search results. In some cases your administrator may have added extra software to make the interface available on the standard HTTPS port.

Contact your system administrator for the access details for your own Funnelback instance.

The main administration interface screen looks similar to the following:

the administration interface 01

The administration interface home screen is composed of several regions, which perform different functions. Some of these are highlighted below:

the administration interface 02
  • Switch betweed dashboards: Access the different administration interface dashboards from this menu. Provides quick access between the marketing dashboard and administration interface.

  • Home links: clicking on the Funnelback logo or the home icon will load the admin UI home screen.

  • Collection and profile switchers: drop down menus containing the collections/profiles available for management by the current logged in user. The profile switcher is only displayed when additional profiles are available to manage.

  • Quick search: search box that allows immediate search against the current selected collection and profile. The search will execute using the default template for the collection.

  • System menus: provides access to collection creation, system and user management and APIs

  • Collection administration: a series of tabs providing various collection management functions. This includes access to template editing, collection configuration, collection update management and usage analytics.

  • Collection overview: dashboard style view providing information and direct links to selected functions on each collection the logged in user has access to manage.

  • Service options: allows setup and deletion of services and provides access to the service view once the current profile is enabled as a frontend service. Enabling as a search services enables front end templates and functionality as well as analytics and optimisation tools via the Funnelback marketing dashboard.

The Funnelback administration interface is currently undergoing a complete rewrite and some functions are now handled by the Funnelback marketing dashboard.

2.1. Marketing dashboard

The marketing dashboard is primarily targeted at non-technical users that are concerned with the reporting functions within Funnelback. This interface also enables users to maintain Funnelback’s best bets, synonyms, curator rules and training data for automated tuning.

The marketing dashboard can be accessed standalone (by accessing https://training-admin.clients.funnelback.com/a/) and will also launch automatically when certain editing functions are selected within the administration interface.

marketing dashboard 01

The marketing dashboard home screen is composed of several features, which perform different functions. These include:

  • User account information and login: clicking on the user name reveals a menu providing access to the user account information (with password change options) and a log out button.

    marketing dashboard 02
  • Service tiles: each frontend service available for management will display as a tile that loads the frontend services control panel when clicked.

  • Services filter: Filters the displayed tiles to only include those that match the text entered within the filter box.

2.1.1. Frontend services dashboard

Selecting a service from the home screen opens a dashboard for managing and analysing the selected service.

services dashboard 01

The service dashboard comprises of the following items:

  • Quick search box: allows a search to be run against the service in preview or live mode. The Funnelback marketing dashboard provides the ability to preview any changes made using the available optimisation tools. This allows best bets, synonyms and curator changes to be made and viewed without the live search being affected. The changes are then published to make them visible on the live search.

  • Display all available services: returns to the frontend services overview screen that lists all of the available frontend services as tiles.

  • Services switcher: quickly switch between available frontend services. Listed frontend services are the same as on the home screen.

  • Show / hide menu bar: clicking on this control hides or shows the menu bar.

  • Return to services dashboard: indicates the current service and clicking returns the user to the service dashboard.

  • Analyse tools: menu of available analysis tools for the current frontend service.

  • Optimise tools: menu of available optimisation tools for the current frontend service.

  • Service summary tiles: are displayed for each analysis and optimisation tool.

The main area of the current service’s dashboard provides access to all the analysis and optimisation tools and displays a tile for each containing a summary of the tool.

2.2. Accessing built-in help

Funnelback includes a searchable manual and context sensitive help available directly from the administration interface.

2.2.1. Contextual help

Each section of the administration interface provides links to context sensitive help screens, either as popups, full page documentation, or both.

Symbol Function

Help icons display contextual help as a tooltip which appears when you hover over the icon with your cursor (in the administration interface) and as a help panel (in the marketing dashboard).

Help

Help links provide a direct link to the online documentation page discussing the feature.

2.2.2. Online manual

A copy of the online manual is included with the Funnelback installation and is available from the help link on the menu bar.

A search of the manual is also provided - enter search terms into the search help box and press enter or click the magnifying glass icon.

online manual 01

2.3. Administering a collection

Common Funnelback administration functions are grouped into separate tabs. The available administration functions differ depending on the permissions applied to your administration user. For example your user may only have the ability to view collection reports - in this case all functions will be greyed out except for those on the analyse tab.

Some functions are also only available after a frontend service is created for the collection.

The administration functions are grouped into:

  • Customise: Quick access to basic configuration. This is where templates, synonyms, best bets and curator rules can be edited.

  • Administer: Access to collection administration functions such as collection configuration, file manager and collection logs.

  • Update: Access to tools to update the collection

  • Analyse: Analytics and audit reports for the collection

  • Tune: Access to the tuning and SEO audit functions

  • Graph: Access to the knowledge graph functions

Exercise 1: Introduction to the administration interface

In this exercise you will be introduced to the main sections available within the administration interface.

  1. Log in to the administration interface by visiting: https://training-admin.clients.funnelback.com/search/admin/

  2. Select the inventors collection from the collection switcher

  3. Swap between the collection menu tabs by clicking their names. The default tab is customise:

    exercise introduction to the administration interface 01
  4. The administer tab provides access to collection-level configuration files, logs and actions:

    exercise introduction to the administration interface 02
  5. The update tab provides options for managing collection updates, schedules and stopping in-progress updates:

    exercise introduction to the administration interface 03
  6. The analyse tab provides access to search usage analytics, collection update history, and reports on various collection metrics:

    exercise introduction to the administration interface 04
  7. The tune tab provides access to the management of tuning test data and tuning results, in addition to content optimisation reports:

    exercise introduction to the administration interface 05
  8. The graph tab provides access to the configuration and management of knowledge graph:

    exercise introduction to the administration interface 06

2.4. Swap between collections and profiles

The collection switcher control is used to select the current collection.

The search preview, tabs and tab options operate on the currently selected collection - so it is very important to ensure that the correct collection is selected before making any further changes.

Exercise 2: Collection and profile switchers

This exercise demonstrates how to switch between collection and profiles.

  1. Log in to the administration interface by connecting to https://training-admin.clients.funnelback.com/search/admin

  2. Use the collection switcher to select the current collection to manage. Change to the inventors collection:

    exercise collection and profile switchers 01
  3. The central region of the screen updates to reflect the inventors collection:

    exercise collection and profile switchers 02
  4. The collection switcher displays I as the active collection

    exercise collection and profile switchers 03
  5. The main title updates to reflect the inventors collection name (Manage inventors)

    exercise collection and profile switchers 04
  6. The search preview box now operates on the inventors collection (and the placeholder text indicates this)

    exercise collection and profile switchers 05
  7. The collection administration and service functions now all operate on the inventors collection.

    exercise collection and profile switchers 06

3. Frontend services

A search collection is often used to underpin a number of different searches that are deployed on a website. These searches may utilise different interfaces, and have different sets of ranking and display options applied to them. A frontend service is defined for each of these different searches.

A frontend service:

  • can cover all, or a definable subset of the content from the collection (e.g. just publications from my website)

  • has one or more templates

  • has independent faceted navigation, best bets, synonyms and curator rules

  • can contain independent display and ranking settings

  • can be tuned independently

  • can have an associated knowledge graph

  • has corresponding analytics that cover the usage of the service.

A frontend service should be created for each collection that is to be searched directly.

Once a frontend service is created it becomes available via the marketing dashboard.

A frontend service is required for:

  • customised templates or web resources

  • configuration of faceted navigation, best bets, synonyms or curator rules

  • configuration of knowledge graph

  • viewing of analytics or audit reports

  • editing tuning training data

Exercise 3: Create a frontend service

This exercise shows you how to enable a frontend service for a profile.

  1. Log in to the administration interface by visiting https://training-admin.clients.funnelback.com/search/admin/

  2. Switch to the inventors collection

    exercise create a service 01
  3. Click the create service button

    exercise create a service 02
  4. View the frontend service by clicking on the view this service in the marketing dashboard button

    exercise create a service 03
  5. If you are prompted for your username and password enter admin for both username and password the click on the log in button.

    exercise create a service 04
  6. The services dashboard screen for the newly created inventors:_default frontend service will be displayed.

    exercise create a service 05
  7. Go to the marketing dashboard home screen by clicking on the service: view all button.

    exercise create a service 06
  8. The marketing dashboard home screen is displayed showing all the available services for the current user. The service name (inventors:_default) and thumbnail image is displayed for each available frontend service. The newly created inventors service is displayed alongside the foodista frontend service.

    exercise create a service 07
Exercise 4: Changing the frontend service name

This exercise illustrates the steps required to change the frontend service name to make it easier to identify for users of the marketing dashboard.

  1. Return to the administration interface by selecting administration from the navigation menu located at the top left hand corner of your screen. Recall that this menu is displayed on all the administration screens and allows quick access to the administration interface and marketing dashboard.

    exercise changing the service name 01
  2. Switch to the inventors collection, and select edit profile configuration from the administer tab

    exercise changing the service name 02
  3. The profile configuraton screen shows any profile settings that are configured for the current profile. At the moment the is a single setting that enables the current profile as a frontend service. This setting was added when you clicked the create service button in the previous exercise.

    exercise changing the service name 03
  4. Define a name for the frontend service by setting the service.title value to Inventors and click save.

    exercise changing the service name 04
  5. Profile settings are staged and can be previewed (by accessing the search using the preview view of the profile. Publish the setting by clicking the publish button that appears on the newly added entry.

    exercise changing the service name 05
  6. Return to the marketing dashboard (click on the view this service in the marketing dashboard button) and observe that the frontend service name is now displayed as Inventors (instead of inventors:_default)

    exercise changing the service name 06
Exercise 5: Changing the thumbnail image for a frontend service

This exercise illustrates the steps required to change the image associated with the frontend service to make it easier to identify for users of the marketing dashboard.

  1. Before we start this exercise download an image to use as a thumbnail. Download http://training-search.clients.funnelback.com/training/training-data/inventors-resources/einstein.jpg to your local machine. We will be using this image as the thumbnail for the service.

  2. Return to the administration interface at https://training-admin.clients.funnelback.com/search/admin/

  3. Switch to the inventors collection, and select browse collection configuration files from the administer tab.

  4. Upload the einstein.jpg image by using the upload control below the inventors / Profile: _default (preview) section. This will upload the selected file to the inventors preview folder.

    exercise changing the thumbnail image for a service 01
  5. Observe that einstein.jpg now appears in the list of files then publish the image file.

    exercise changing the thumbnail image for a service 02
  6. Switch to the profile configuration editor and define the thumbnail image by adding service.thumbnail and setting this to einstein.jpg to the file then click the save button or icon. Note: the file name is case sensitive.

    exercise changing the thumbnail image for a service 03
  7. Publish the einstein.jpg by clicking the publish button on the right hand side of the row.

  8. Return the marketing dashboard home and observe that the inventors frontend service now includes a thumbnail image

    exercise changing the thumbnail image for a service 04

4. Funnelback templating

The search results produced by Funnelback are fully customisable. This allows the search results to be adapted for any HTML design or text-based formats such as CSV, XML and JSON.

Funnelback uses the Freemarker templating language to define the search templates. The templates look like HTML files with additional custom tags - anyone familiar with editing HTML will be able to understand the Freemarker template code. The Freemarker documentation should be read to assist in understanding the Funnelback templates.

A single search results template in Funnelback must handle two cases covering what to present to the end user before and after a search is run.

4.1. Template preview and live modes

The Funnelback administration interface provides the ability to preview changes made to template files, allowing changes to be made and viewed without the live search being affected.

The changes are then published to make them visible on the live search.

Exercise 6: Preview and publish template changes

This exercise looks at a bare bones search results template and relates the template file to what you see on screen.

  1. Log in to the administration interface by visiting https://training-admin.clients.funnelback.com/search/admin/

  2. Switch to the inventors collection.

  3. From the administration interface, view the template in preview mode. Select the preview radio button then click on the arrow icon. This runs the search and opens the results in a new browser tab.

    exercise preview and publish template changes 01
  4. The search results are loaded in a new browser tab - a very basic search box is displayed - this is the screen that displays when no query is present. Observe that the URL (https://training-admin.clients.funnelback.com/s/search.html?collection=inventors&profile=_default_preview) includes two URL parameters - collection and profile. The Funnelback query processor is accessed by visiting the search.html URL and passing parameters to it that tell Funnelback what collection and profile (live or preview) to use.

    exercise preview and publish template changes 02
  5. View the HTML page source in your web browser. Note that the code returned includes only the HTML form code to call the Funnelback search with basic parameters.

    exercise preview and publish template changes 03
  6. Close the view source browser and tab containing the search results screen. Preview the search again, this time entering invention into the search box before submitting the query. Observe that the URL called (https://training-admin.clients.funnelback.com/s/search.html?collection=inventors&profile=_default_preview&query=invention) is the same as the previous URL, but with one additional parameter, query. This time a search results listing is returned because a query was included in the URL. This demonstrates that the search results template handles two types of result screens - a screen (normally containing a search box) when no query is present and a screen containing a search results listing when a query has been provided.

    exercise preview and publish template changes 04
  7. Return to the administration interface by closing the tab. Load the template manager by selecting edit result templates from the customise tab.

    exercise preview and publish template changes 05
  8. The template management screen allows you to edit, rename, delete, publish and unpublish existing templates and create or upload new templates for use with the current frontend service. Edit the template by clicking on the simple.ftl link. simple.ftl is the default template.

    exercise preview and publish template changes 06
  9. The template file loads in the built-in template editor:

    exercise preview and publish template changes 07

    If you are familiar with HTML you will probably be able to read the template and get a sense of what it’s doing. Freemarker templates are made up of XML-style tags that look like <@ MACRO > (Freemarker macro) or <# DIRECTIVE > (Freemarker directives), and Freemarker variables that look like ${ VARIABLE }. Everything else in the template will be returned as is. The HTML code inside the <@s.AfterSearchOnly> macro is rendered only after a search is run. This explains why the first preview of the search only displayed a search box, but after a search for inventors was run the search results were also displayed. It also means that the same template file is used to provide the interface for both use cases - before and after a search has taken place.

  10. Add a heading image to the results page. Locate the <body> tag and insert:

    <div><img src="http://training-search.clients.funnelback.com/training/training-data/inventors/media/images/women_header.jpg" ></div>

    immediately after the tag.

  11. Save your changes by clicking the save button (but don’t save and publish at this stage).

    exercise preview and publish template changes 08
  12. Return to the administration home and preview your template again - view the search before and after submitting a query to view the effects of your change.

  13. View the template again, but select the live radio button before submitting the search. Observe that the banner image is not displayed. This is because the change has been made only to the preview mode of the search - to make the change live the template needs to be published.

  14. The template management screen (accessed via the edit template button on the customise tab) indicates that the template has unpublished changes. The file listing also shows a backup file that was automatically created when you saved the template when you toggle the show backup files option below the file listing.

    exercise preview and publish template changes 09
  15. Publish the template by clicking on the publish button. The status will update to published.

    exercise preview and publish template changes 10
  16. View the live mode of the search results page again and confirm that the banner is now displayed. Select live from the view menu to show the published (live view) templates then click the small eye icon on the right hand side of the simple.ftl row to load the search with the live version of the template.

4.2. Debugging template errors

Funnelback logs user interface issues to the modernui log. These are accessible via the log viewer from the collection log files for the collection. The relevant log files are:

  • modernui.Public.log contains errors for searches made against the public HTTP and HTTPS ports.

  • modernui.Admin.log contains errors for searches made against the administration HTTPS port.

Funnelback also includes some settings that allow this behaviour to be modified so that some errors can be returned as comments within the (templated) web page returned by Funnelback.

The following configuration options can be set on your collection’s configuration while you are working on template changes:

ui.modern.freemarker.display_errors

Specifies whether or not to return Freemarker errors within templated source code.

ui.modern.freemarker.error_format

Specifies the formatting that should be used when Freemarker errors are returned.

The error format can also be set to return as HTML or JSON comments.

The string error format will return the error as raw text, which results in the error message being rendered (albeit unformatted) when you view the page - this is the recommended option to choose while working on a html template as the errors are not hidden from view.

Setting these causes the template execution to continue but return an error message within the code.

Syntax errors in the template resulting in a 500 error are not returned to the user interface - the browser will return an error page.

4.2.1. Data model log object

The data model includes a log object that can be accessed within the Freemarker template allowing custom debug messages to be printed to the modern UI logs.

The log object is accessed as a Freemarker variable and contains methods for different log levels. The parameter passed to the object must be a string.

The default log level used by the modern UI is INFO. It is not possible to change this on a per-collection level and changing the level globally requires administrative access.

This means when debugging your templates you will probably need to print to the INFO level - but don’t forget to remove your logging once you have completed your testing otherwise a lot of debugging information will be written to the log files for every single query.

e.g.

<#-- print the query out at the INFO log level -->
${Log.info("The query is: "+question.query)}
<#-- print the detected origin out at the DEBUG log level -->
${Log.debug("Geospatial searches are relative to:  "+question.additionalParameters["origin"]?join(","))}
Messages written via the log object will only be logged when accessing the search.html endpoint.
Exercise 7: View template error messages

This exercise examines the log files used to hold errors generated by templates.

  1. Log in to the administration interface by visiting https://training-admin.clients.funnelback.com/search/admin/

  2. Switch to the inventors collection.

  3. From the administration interface, select the customise tab and click on the edit result templates button then edit the simple.ftl.

  4. Add the following code above the <!DOCTYPE html> line then save the template. This introduces an unclosed variable to the template which will break the display.

    ${broken
    exercise view template error messages 01
  5. View the broken template by conducting a search for invention (https://training-admin.clients.funnelback.com/s/search.html?collection=inventors&query=invention&profile=_default_preview). Observe that an error is now returned.

    exercise view template error messages 02
  6. Return to the administration interface and select browse log files from the administer tab. Template errors are written to the following logs, which appear beneath the collection logs heading:

    • modernui.Public.log: This log is used when accessing the search results via the public HTTP or HTTPS ports (by default this is for http:// or https:// requests to the search that use standard ports) For the training environment requests made to http://training-search.clients.funnelback.com and https://training-search.clients.funnelback.com:8081 are logged here.

    • modernui.Admin.log: This log is used when accessing the search results via the admin HTTPS url (normally on port 8443). For the training environment this is requests made to $https://training-admin.clients.funnelback.com`. This URL is normally only accessed when you preview a search from the administration interface.

    If you are using the Windows version of Funnelback the modernui log files are found within the system logs section, found by selecting view system logs from the system menu. The system logs screen is divided into system logs and web logs.

    exercise view template error messages 03

    The logs are prefixed with the collection ID and appear beneath the web logs heading. E.g. the equivalent logs under windows would be named: modernui.inventors.Public.log and modernui.inventors.Admin.log.

  7. View the tail of the modernui.Admin.log file (as the search has been launched from the administration interface search box) by clicking on the show last 100 button for the log file.

  8. The end of the log file should provide clues about what caused the error. If the start of the error message is missing the number of lines returned can be increased by editing the tail parameter in the URL (eg. tail=200 will return the last 200 lines of the log file). The log file indicates that the template contains an error at line 11.

    2019-11-06 03:16:56,462 [admin-2708] [inventors:_default_preview] ERROR interceptors.FreeMarkerParseExceptionInterceptor - Error parsing FreeMarker template freemarker.core.ParseException: Syntax error in template "conf/inventors/_default_preview/simple.ftl" in line 11, column 5:
    Encountered ">", but was expecting one of:
        "."
        ".."
        <DOT_DOT_LESS>
        "..*"
        "?"
        "??"
        "!"
        "["
        "("
        "}"
        <TERMINATING_EXCLAM>
    	at freemarker.core.FMParser.generateParseException(FMParser.java:6107)
    	at freemarker.core.FMParser.jj_consume_token(FMParser.java:5966)
    	at freemarker.core.FMParser.StringOutput(FMParser.java:1670)
    	at freemarker.core.FMParser.MixedContentElements(FMParser.java:3955)
    	at freemarker.core.FMParser.Root(FMParser.java:4652)
    	at freemarker.template.Template.<init>(Template.java:252)
    	at freemarker.cache.TemplateCache.loadTemplate(TemplateCache.java:549)
    	at freemarker.cache.TemplateCache.getTemplateInternal(TemplateCache.java:439)
    	at freemarker.cache.TemplateCache.getTemplate(TemplateCache.java:292)
    	at freemarker.template.Configuration.getTemplate(Configuration.java:2798)
    	at freemarker.template.Configuration.getTemplate(Configuration.java:2656)
    	at org.springframework.web.servlet.view.freemarker.FreeMarkerView.getTemplate(FreeMarkerView.java:373)
    ...
  9. Return to the template editor and correct the error by closing the bracket on the variable added earlier in the exercise before saving the template. If you saved and published in the earlier step make sure you save and publish again now to ensure that the change is applied to both the preview and the live versions of the template.

    ${broken}
  10. Rerun the search. The browser still shows an error, however this time the error is due to the variable being undefined and there is some text providing a clue as to the problem. This error can be investigated in the same way as the previous error.

Exercise 8: Enable template debugging

In this exercise the collection configuration will be updated to return errors to the template as plain text.

Note: this approach only works on some types of errors - debugging using log files is still required if the template fails to compile.
  1. Select edit collection configuration from the administer tab.

  2. Check to see if ui.modern.freemarker.display_errors is listed in the currently set keys

    exercise enable template debugging 01
  3. Add the setting by clicking the add new button. Start typing freemarker iinto the parameter key field and select ui.modern.freemarker.display_errors from the suggested entries.

    exercise enable template debugging 02
  4. Set the value to true then save the setting.

    exercise enable template debugging 03
  5. Repeat to set ui.modern.freemarker.error_format.

    The error format can be set to one of exception, json, html or string. Exception (the default) causes template processing to halt completely if an error is encountered (which will usually result in either a blank screen or a server error page being returned). JSON and HTML return the error messages wrapped in JSON or HTML comments. Set the value to string, which returns the error as a string, so that the errors will appear in the browser display.

    exercise enable template debugging 04
  6. Rerun the query for invention observing the error message that is returned and also that the search results are now returned below the error message.

    exercise enable template debugging 05
  7. View the page source to read the error message with some formatting - as the error message is currently returned as raw text the browser will join all the text into a single paragraph.

    exercise enable template debugging 06
  8. The two collection configuration settings should be removed once testing of the template is complete to avoid errors being exposed to users. Return the the edit collection configuration screen and remove the two ui.modern.freemarker options added above.

  9. Reset the template by removing the ${broken} parameter that was added above.

  10. Print some custom log messages. Edit the simple.ftl and add the following lines after the <@AfterSearchOnly> tag on (approx.) line 30:

    <#-- print the query out at the INFO log level -->
    ${Log.info("The query is: "+question.query)}
    <#-- print the detected origin out at the DEBUG log level -->
    ${Log.debug("Geospatial searches are relative to:  "+question.additionalParameters["origin"]?join(","))}
  11. Run a search against the inventors collection and observe the messages written to the modernui.Public.Log (or modernui.Admin.log). Observe that only the INFO level message is displayed. This is because the default log level is INFO (meaning deeper log level messages are suppressed). Refer to the previous exercise if you can’t remember how to access the log files.

  12. Modify the simple.ftl and amend the geospatial log call to use the INFO log level and observe the messages in the modernui logs.

  13. Reset the template by removing the two Log calls.

Extended exercise: template debugging
  1. Experiment with the effect of changing the error format on both the template and log files.

4.3. Template editor shortcuts

The built-in template editor has a number of keyboard shortcuts that can assist with editing.

Windows Macintosh

Save (but don’t publish) changes

Ctrl+S

control ⌃+S

Jump to a line number

Ctrl+L

command ⌘+L

Find (click + button on find popup for replace)

Ctrl+F

command ⌘+F

Note: the jump to line option only works in some browsers.

4.4. Customising the search results block

Search results are templated according to the code that appears within the <@s.Results> block of code.

<@s.Results>
<#if s.result.class.simpleName != "TierBar">
CODE FOR EACH RESULT
</#if>
</@s.Results>

Basic results item variables available:

  • s.result.title: the result title

  • s.result.summary: the result summary

  • s.result.liveUrl: the URL of the target document

  • s.result.clickTrackingUrl: the URL to use for link actions to make use of Funnelback’s click tracking feature.

  • s.result.cacheUrl: the URL to use to access the cached version of the document

  • s.result.date: the date associated with the document.

  • s.result.fileType: the file type of the document

  • s.result.rank: the position (rank) of the current result item inside the result set.

  • s.result.listMetadata: map of metadata elements available for use in templates. Values for each are returned as a list of strings. Includes only metadata fields that have been configured (using display options) to be returned with the search results.

  • s.result.metaData: map of metadata elements available for use in templates. Values for each are returned as a string with fields delimited using a veritical bar charackter. Includes only metadata fields that have been configured (using display options) to be returned with the search results.

To print any of these values wrap them in the Freemarker directive ${VARIABLENAME}. E.g. ${s.result.title} will be replaced with the title text for the current result.

Before accessing any variable ensure that the variable being printed is defined:

  • by wrapping the variable in an <#if> statement using the missing value operator ??

    <#if s.result.title??>${s.result.title}<#else>No title</#if>
  • or for cases where you are printing variable without surrounding markup an exclamation mark (!) can be used e.g ${VARIABLENAME!}

    • ${s.result.title}: template returns an error if the variable is not defined.

    • ${s.result.title!}: prints variable if set otherwise nothing

    • ${s.result.title!"Empty"}: prints variable if set otherwise prints Empty or any string you define.

Exercise 9: Customise the search results listing

This exercise shows how to make some simple changes to the formatting of the individual search results.

  1. Log in to the administration interface by visiting https://training-admin.clients.funnelback.com/search/admin/

  2. Switch to the inventors collection.

  3. Edit the template for the by selecting design results page from the customise tab then clicking on the simple.ftl link. Locate the results block (the code between <@s.Results> and </@s.Results>, starting at approx line 97). Observe that the ordered list container appears outside the <@s.Results> tags. The Freemarker code inside the results block is expanded for each result item in the result set. Variables, such as ${s.result.title} are replaced with the value from the current result item. For example ${s.result.date} is replaced with the date of the current result.

    exercise customise the search results listing 01
  4. Modify the template so that the search results are returned inside a table. When you are finished save the change (but don’t publish yet). Hint: you need to replace the <@s.Results> block and surrounding <ol> tags and replace with the following:

    <table>
      <tr><th>Title</th><th>URL</th><th>Summary</th></tr>
      <@s.Results>
        <#if s.result.class.simpleName != "TierBar">
          <tr>
            <td>
              <a href="${s.result.clickTrackingUrl}" title="${s.result.liveUrl}">
                <@s.boldicize><@s.Truncate length=70>${s.result.title}</@s.Truncate></@s.boldicize>
              </a>
            </td>
            <td>
            <cite><@s.cut cut="http://"><@s.boldicize>${s.result.displayUrl}</@s.boldicize></@s.cut></cite>
            </td>
            <td>
            <#if s.result.summary??>
              <p>
                <#if s.result.date??>${s.result.date?date?string("d MMM yyyy")}:</#if>
                <@s.boldicize><#noautoesc>${s.result.summary}</#noautoesc></@s.boldicize>
              </p>
            </#if>
            </td>
          </tr>
        </#if>
      </@s.Results>
    </table>
  5. Preview your changes. You should see the results reformatted in a table, similar to what’s displayed below. If you receive a server error then check to make sure you haven’t accidently broken any of the macro or conditional Freemarker tags.

    exercise customise the search results listing 02
Extended exercise: result customisation
  1. Experiment and make some other changes to the template to change the result formatting. Try displaying some of the other items listed under the customising the search results block heading above. For example add the result rank as an additional first column.

4.5. The data model - an introduction

Behind every search page is a highly detailed data structure containing all the data relating to the search transaction. This data structure, known as the data model, contains information relating to the search query, response and configuration. Every element that is returned in the data model can be utilised by the search template.

4.5.1. Top-level structure

The Funnelback data model consists of two main objects:

  • question: includes all of the data defining the search question. This includes the search keywords, the collection, configuration, ranking and display options, and other search constraints.

  • response: the response contains all of the information returned by the Funnelback query processor in response to the question object. This includes the search results and associated metadata, faceted navigation and result counts, spelling suggestions, best bets etc.

Exercise 10: Viewing the data model for a search as XML or JSON

It is often useful to view the underlying data model for a query when building a search template or debugging search behaviour. The data model shows you all the settings that are applied and also all of the data elements and values that you can use within the search template. This exercise shows how to view the underlying data model for a Funnelback query in XML or JSON.

A browser plugin or extension for formatting of JSON (such as JSONView) may be required to view the JSON response.
  1. Log in to the administration interface by visiting https://training-admin.clients.funnelback.com/search/admin/

  2. Switch to the inventors collection

  3. Run a search for invention: enter invention into the search preview box in the administration interface and execute the search.

    exercise viewing the data model for a search as xml or json 01
  4. Edit the URL in your browser and change search.html to search.xml to view the data model behind the search for invention as XML. This is the XML endpoint for the search and shows the underlying data structure for the query for invention that you just ran.

    exercise viewing the data model for a search as xml or json 02
  5. Edit the URL in your browser and change search.xml to search.json to view the data model behind the search for invention as JSON. Observe that the data structure is the same as the one that you just viewed as XML. If your browser just displays unformatted text then you don’t have a JSON viewer browser extension installed on your computer. If you are seeing unformatted text install a browser extension to enable you to view the JSON.

    exercise viewing the data model for a search as xml or json 03

    Inspect the question object. Observe that there are sub-items, and that each item contains a key and value component.

    The value component indicates the data type of the field. Funnelback fields are one of the following types:

    • String: indicated by double quotes e.g. title: "example"

    • Number: indicated by an integer without quotes e.g. rank: 0

    • Sequence: (array/list) indicated by square brackets e.g. selectedFacets: []

    • Hash: (map/associative array) indicated by curly brackets e.g. environmentVariables: {}

    While the XML endpoint will work in most browsers, it is recommended that you install a suitable JSON viewer plugin and user the JSON endpoint. The XML is useful for viewing the structure but doesn’t provide you with information on the different variable types so is only of limited use when viewing the data model.

    The data type is important as Freemarker has different built-in functions that operate on the variable depending on the variable’s type.

    Clicking on an item within the data model picker will insert a variable into the template editor at the current cursor location.

    The key is case-sensitive so take care when accessing the variables from Freemarker templates.

4.6. Freemarker macro libraries

Funnelback ships with two libraries of Freemarker macros, providing quick pre-built functions that can be used to build your search interface.

The libraries are loaded into the search template using <#import> directives located at the top of the template file.

freemarker macro libraries 01

The two import directives above import the funnelback_classic and funnelback macro libraries into the s and fb namespaces respectively.

The macros included within these libraries can be accessed once imported by prefixing the macro call with the namespace. The libraries contain functions that implement conditional logic (e.g. whatever appears inside this tag is run only after a search keyword is entered) or handle logic required to implement the display of a feature.

E.g. <@s.Results> calls the Results macro from the s namespace (which corresponds to the funnelback_classic macro library).

  • funnelback_classic: contains macros providing access to Funnelback’s core functionality.

  • funnelback: contains macros providing access to extended and newer functionality.

Funnelback documentation

4.7. Freemarker built-in functions

Freemarker has an extensive set of built-in functions that can be called when a value is printed. A built-in function is called by appending a question mark (?) to the variable name and adding the function. E.g. ${s.result.title?upper_case} prints the result title as an uppercase string. Function calls can also be chained. ${s.result.title?upper_case?html} will convert the title to upper case then html encode it.

The built-in functions provide a rich set of tools that can be used to manipulate or transform the variables allowing a huge amount of flexibility to be built in to the search result templates.

When working with Freemarker it is important to know what version of Freemarker is used by the version of Funnelback you are using as some built-in functions are not available in older versions of Freemarker.

4.7.1. Variable escaping

Freemarker has built-in support for the automatic escaping of variables that are in templates. Escaping is an important tool that ensures template generated-code is valid and is also an important security feature as correct use of escaping mitigates some security issues such as injection style attacks.

Templates should define an output format that specifies what type of content is being returned. The defined output format will apply escaping rules to all variables contained within the template unless you choose to override the default escaping for a section of the template or when expanding specific variables.

4.8. Creating additional templates

Multiple templates can be configured for each Funnelback search. Different templates may be used to format results for different use cases that require different formatting (e.g. general search, publications search) or to return the results in other formats (e.g. return the results formatted as a CSV file).

The default template for any collection or service in Funnelback is the simple.ftl template. Additional templates with arbitrary file names can be defined.

Search results templates in Funnelback usually have the following overall logic:

if (search has not been performed)
    display empty search input box and branding
else
    display a search refinement box
    if (search has matching results)
        for each matching result
            display result
    else
        display zero results message
Exercise 11: Create a custom template

In this exercise a custom template will be created and configured to provide some basic logic for the presentation of the search results.

  1. Log in to the administration interface by visiting https://training-admin.clients.funnelback.com/search/admin/.

  2. Switch to the inventors collection. Open the template management screen by selecting edit templates from the customise tab.

    exercise create a custom template 01
  3. Click the add new button to create a new template.

    exercise create a custom template 02
  4. Enter the name mytemplate.ftl into the create new file popup then click save.

    exercise create a custom template 03
  5. This will create a new empty template file and add it to the file listing. Edit the template by clicking on the newly created mytemplate.ftl link.

    exercise create a custom template 04
  6. Cut and paste the code below into the editor window then click save and publish. This will create a new template called mytemplate that implements the logic described above.

    <#ftl encoding="utf-8" output_format="HTML"/>
    <#import "/web/templates/modernui/funnelback_classic.ftl" as s/>
    <#import "/web/templates/modernui/funnelback.ftl" as fb/>
    <!DOCTYPE html>
    <html lang="en-us">
    <head>
    <title>My search template</title>
    </head>
    <body>
     <#-- NO QUERY -->
    <@s.InitialFormOnly>
        <p>No query submitted yet.  Sorry.</p>
    </@s.InitialFormOnly>
    <@s.AfterSearchOnly>
        <#-- DISPLAY RESULTS -->
        <ol>
        <@s.Results>
            <#if s.result.class.simpleName == "TierBar">
                 <#-- TIER BAR FOR PARTIAL MATCHES -->
                <#if s.result.matched != s.result.outOf>
                    <p>Search results that match ${s.result.matched} of ${s.result.outOf} words</p>
                </#if>
            <#else>
                <li><a href="${s.result.clickTrackingUrl}" title="${s.result.liveUrl}">${s.result.title}</a></li>
            </#if>
        </@s.Results>
        </ol>
        <#-- NO MATCHES -->
        <#if response.resultPacket.resultsSummary.totalMatching == 0>
            <p>Sorry.  No matches found.</p>
        </#if>
    </@s.AfterSearchOnly>
    </body>
    </html>
  7. Return to the template manager by clicking the back to directory link. Preview the custom search template’s initial state by locating the mytemplate.ftl file in the listing and clicking the preview button (icon which looks like an eye). Observe that the URL for the search includes a parameter form=mytemplate. The form parameter tells Funnelback which template should be applied to the search results when running the search query.

    exercise create a custom template 05
  8. Run a test query using the new template. Edit the URL and add &query=barbie to the URL used to preview the template. Press enter and the search results display should update to present a list containing two search results.

    exercise create a custom template 06
  9. Define a query that will return no search results: Replace the query=barbie in the URL with query=qwerty.

    exercise create a custom template 07
Extended exercises: conditional templating
  1. Define a query that will display partially matching results (results that match some of the words in the query). Replace query=qwerty in URL with query=hair dryer.

  2. Update the text that is presented when there are no search results.

  3. Experiment with some of the Freemarker built-in functions and observe the effect on the search results output.

4.9. Web resources

Most templates rely on external resources such as image, JavaScript and CSS files.

These external resources can be stored at web accessible locations such as your public web server and referenced as external resources from the template using absolute URLs.

Funnelback provides web resource folders that can also be used for local storage of these files. Web resource folders are associated with the folders that contain the search template files.

The URL of a web resources folder is constructed using the following pattern: <PROTOCOL>://<FUNNELBACK-SERVER>/s/resources/<COLLECTION-ID>/<PROFILE-ID>/<FILENAME>

where:

  • <PROTOCOL> is either http or https

  • <FUNNELBACK-SERVER> is the hostname of the Funnelback server. If referencing from a template the <FUNNELBACK-SERVER> and <PROTOCOL> can be omitted and presented as a relative link.

  • <COLLECTION-ID> is the collection containing the resource. (This can be copied from the collection parameter in the URL of the file manager). When accessing a resource from within a search template <COLLECTION-ID> should be replaced with

    ${question.collection.id}
  • <PROFILE-ID> is the profile containing the resource (this can be copied from the profile parameter in the URL of the file manager). When accessing a resource from within a search template <PROFILE-ID> should be replaced with

    ${question.profile}
  • <FILENAME> is the filename of the web resource.

    E.g. a link to a resource from a template example.css might be:

    <link rel="stylesheet" href="/s/resources/${question.collection.id}/${question.profile}/example.css" />
It is often a good idea to use protocol-independent URLs when linking to web resources to avoid mixed content warnings when working with HTTPS urls.
Exercise 12: Web resources

This exercise shows how to manage your web resources and access them from your Funnelback templates.

  1. Log in to the administration interface by visiting https://training-admin.clients.funnelback.com/search/admin/

  2. Switch to the inventors collection.

  3. Select Edit web resources files from the customise tab.

  4. Create a CSS file containing the relevant style rules. Download http://training-search.clients.funnelback.com/training/training-data/inventors-resources/inventor-styles.css to your local machine.

  5. Click the upload files button at the bottom of the web resources screen to upload the inventor-styles.css file that you just downloaded.

    exercise web resources 01
  6. Observe that inventor-styles.css is now listed, but has a status of new. Web resources can be previewed allowing changes to me made to web resources that can be tested without being live.

    exercise web resources 02
  7. Modify the default search template (simple.ftl) to reference the newly added CSS file. Remember that the resource will only be available from the preview version of the template until both the template and the CSS file are published. Add the following to the document’s <head> and click save.

    <link rel="stylesheet" href="/s/resources/${question.collection.id}/${question.profile}/inventor-styles.css" />
  8. Test the search by running some queries and verify that the linked styles are now being applied.

    exercise web resources 03
  9. Publish both the CSS and template files and verify that the changes are live.

  10. Make the CSS file available from a css subfolder. Hint: remove the css file the create a css folder (using the add new button) then upload the css file into the newly created folder. Update your search template to read the CSS file from the new location.

Extended exercises: style the results page
  1. Style the results page for the inventors search to match the styling for the inventors website. This will require additional styles to be defined within the imported stylesheet and additional markup (<div> tags etc.) to be added to the main results template. 

4.10. Using WebDAV to maintain templates and web resources from your desktop

Funnelback can be mounted as a network drive using a compatible WebDAV client. This exposes the template and web resources folders and allows maintenance and editing of the files directly from your desktop computer.

Once mounted you can perform a number of tasks including:

  • Edit templates using a local text editor of your choice.

  • Create new templates.

  • Upload web resources using drag and drop.

  • Create a folder structure beneath the web resources folder.

Exercise 13: Mount Funnelback using WebDAV

Thie exercise shows how to access your Funnelback templates and web resources directly from your desktop. For this exercise you will need to install Cyberduck. Cyberduck is used in this example as it works on both Windows and MacOS.

  1. Download and install Cyberduck

  2. Open Cyberduck.

  3. Create a new bookmark by selecting new bookmark from the bookmark menu.

  4. The bookmark configuration screen opens. Select WebDAV (HTTPS) from the dropdown menu.

  5. Enter the following details into the bookmark configuration screen:

    • Server: training-admin.clients.funnelback.com (e.g. search.mycompany.com)

    • Port: 443 (e.g. 8443)

    • Username: your administration interface username (e.g. jsmith)

    • Path: /admin-api/file-management/v1/webdav/

      exercise mount funnelback using webdav 02
  6. Save the bookmark

  7. Double click on the bookmark to connect to the server. If you receive a certificate warning you can ignore this (if you are using the Vagrant training VM it uses a self-signed SSL certificate). You will be prompted for a password on your first connection attempt. Ensure you check the save password / add to keychain option before continuing.

    exercise mount funnelback using webdav 03
  8. The file window will display, showing collections at the root. Expanding this will show all the collections that you have access to, with subfolders for each profile, then live and offline views, and then templates and web resources.

    exercise mount funnelback using webdav 04
  9. Navigate around the folder structure. You will see a top level collections folder. Beneath this you should see all the collections that your user has access to. Within each of these folders is a folder for each profile that your user has access to. Inside this folder you will find live and offline folders which correspond to the live and offline profile views. Beneath this is where you will find the template and web resource folders. The live folder can be viewed but the contents are read only.

Exercise 14: Edit a template using WebDAV

This exercise demonstrates how templates can be edited in a locally installed text editor.

  1. Edit the mytemplate.ftl template for the inventors collection. Browse to the collections > inventors > profiles > default > preview > templates folder. Right click on the mytemplate.ftl then select edit with from the popup menu and choose a suitable text editor.

    exercise edit a template using webdav 01
  2. The file will open in your text editor. Add a title immediately before the <ol> tag: <h1>Title added via WebDAV</h1> then save the file. This will update on the Funnelback server. Note: when you save a file this way it overwrites the file on the server without creating an automatic backup of the file. Changes will be reflected when viewing the file on the Funnelback server. If you wish to publish the file you will need to do this from the admin interface.

    exercise edit a template using webdav 02
  3. Open the administration interface, select the inventors collection then edit templates. Observe that unpublished changes are recorded for the mytemplate.ftl file. Click on this and observe that the text you just added and saved in your text editor is showing inside the built-in editor.

    exercise edit a template using webdav 03
    exercise edit a template using webdav 04
  4. View the template by running a search using that template and observe that the title is displaying. http://training-search.clients.funnelback.com/s/search.html?collection=inventors&profile=_default_preview&form=mytemplate&query=test

  5. The file will still need to be published using the administration interface as the live folder when mounted via WebDAV is read only.

Exercise 15: Add web resources using WebDAV

In this exercise you will manage your web resources directly from your desktop.

  1. Download this image and save it locally http://training-search.clients.funnelback.com/training/training-data/inventors-resources/Professor-Frink.jpg.

  2. Navigate and click into the web-resources folder for the inventors collection, again beneath a preview profile.

  3. Observe that the css folder that you created via the admin interface is displayed. Create a new folder called mystuff (inside the same folder as the css folder) by selecting new folder from the menu.

    exercise add web resources using webdav 01
  4. Navigate into this folder and drag the image you just saved into this folder.

    exercise add web resources using webdav 02
  5. Return to the administration interface and select edit web resources from the customise tab for the inventors collection.

  6. Observe that the mystuff folder you created is displayed as a subfolder to the web resources, and that the file you just uploaded is also present.

    exercise add web resources using webdav 03
    exercise add web resources using webdav 04
  7. The folder and file will still need to be published using the administration interface as the live folder when mounted via WebDAV is read only.

5. Best bets

A best bet is used to provide additional featured items independent to the set of search results.

Best bets are completely independent of the search index and can link to items that are not part of the websites that make up the search. They can also be used for promotional purposes to promote a certain page or document based upon a user’s query.

best bets 01

The style and appearance of the best bets in the search results is governed by the style sheets that are applied to the search results template, and the position can be controlled by a search administrator with the ability to edit the templates.

5.1. Managing best bets

Best bets are managed from the marketing dashboard. The best bets manager provides tools for creation, editing, cloning and deletion of best bets, and also the ability to publish and unpublish.

If the service does not have any best bets defined the following screen is displayed when accessing the best bets section:

managing best bets 01

If there are best bets defined a table listing the best bets is displayed. Clicking on a best bet opens the best bet inside the editor. Administrators also have the ability to publish unpublish, clone and delete a best bet

managing best bets 02

Cloning an item makes a copy of an existing best bet that can then be edited.

Best bets are not available in the live search until they are published. This allows a best bet to be created and tested before release, or staged for later use.

Clicking the add new button opens the best bet editor.

5.1.1. Creating and editing best bets

Best bets are configured using a simple editor screen.

The example below shows the configuration for the best bet that is shown above.

creating and editing best bets 01

Each best bet requires the following:

  • Title: used for the hyperlinked text for the best bet result

  • Description: used for the summary text presented below the title. This can include HTML formatting.

  • URL: this is the URL to link to when the best bet is clicked on. The URL can be any URL and does not need to be part of the search. There is an option that allows the URL to be removed from the set of search results if it matches the URL for the best bet.

  • Trigger: these are the search terms that will cause the best bet to be displayed.

  • Trigger type: this controls how Funnelback compares the user’s query to the best bet trigger. There are four types of triggers:

    • The search keyword(s) exactly matches: This will only trigger if the user’s keyword is identical to the trigger.

    • All words must be present in the search keyword(s), in any order: This is the most commonly used trigger and matches when all the trigger terms appear within the user’s query. E.g. a best bet using a trigger of red wine will appear as long as the words red and wine both appear somewhere in the user’s query.

    • Substring match: The best bet is returned if the trigger is a substring of the user’s query. E.g. a best bet with a trigger of red will be returned for the following queries: red wine, reduction, blackened redfish.

    • Regular expression match: The best bet is returned if the trigger regular expression matches the user’s query. This is an advanced match type for power users allowing advanced matching such as wildcards. If you are unfamiliar with regular expressions then don’t use this trigger type.

5.1.2. Previewing and publishing

Funnelback provides the ability to preview changes made to best bets and other configuration allowing changes to be made and viewed without the live search being affected.

The changes are then published to make them visible on the live search.

Best bets once live can also be unpublished - this removes them from the live search and allows them to be previewed and edited for future use.

When an item is saved but marked as unpublished it can be viewed by using the search box located at the top of the marketing dashboard and ensuring the preview option is chosen from the drop down.

previewing and publishing 01

This will run a search using the live index, but apply anything that is marked as unpublished. This allows a best bet to be created and tested before it is released.

Selecting live from the search box will run the search against the live index applying only configuration that has been published and is equivalent to what public users of the search will see.

Exercise 16: Creating a best bet

In this exercise you will create a simple best bet and see how it is returned in the search results.

  1. Log in to the administration interface and switch to the inventors collection

  2. Select customise best bets from the customise tab in the administration interface. If this option is greyed out then the selected collection/profile combination is not set up as a service. This needs to be done before best bets can be edited. If so click the green create service button to designate this collection/profile combination as a service.

    exercise creating a best bet 01
  3. The marketing dashboard will open on the best bets screen.

    exercise creating a best bet 02
  4. You can also access the best bets editor from within the marketing dashboard by either selecting it from the left-hand menu of the dashboard or via the default dashboard screen of your service.

    exercise creating a best bet 03
  5. Click the add new button to open the best bets editor. Observe that the preview is updated dynamically as data is entered into the form. Enter the following into the best bets:

    • Trigger keywords: toys

    • Match type: The search keyword(s) exactly matches

    • Title: Barbie dolls - Ruth Handler

    • URL to display/link to: http://training-search.clients.funnelback.com/training/training-data/inventors/Ruth-Handler.html

    • Description: Perhaps one of the most famous toys in American history, the Barbie doll is a staple in the toy chests of little girls everywhere. Along with co-founding the renowned toy company Mattel, woman inventor Ruth Handler also designed the doll that would become an American cultural icon.

    exercise creating a best bet 04
  6. Click the add button (not add and publish) to create the best bet.

  7. Observe that the new best bet now appears in the list of best bets, and that it has a status of new and that there is a button available to publish the best bet.

    exercise creating a best bet 05
  8. Run a search for toys using the search box at the top of the marketing dashboard. Ensure that preview is selected from the drop down.

    exercise creating a best bet 06
    exercise creating a best bet 07
  9. Observe the best bet appearing above the search results. Return to the marketing dashboard and run the query again, this time ensuring that live is selected from the drop down.

    exercise creating a best bet 08
  10. Observe that the same results are returned, but the best bet is not displayed. This is because the best bet has not yet been published. Return to the best bets editor and publish the best bet, observing that the status now changes to published, and that the unpublish button is now clickable. Clicking unpublish will return the best bet to the new / unpublished state.

    exercise creating a best bet 09
  11. Re-run the query ensuring that live is selected from the drop down menu. Observe that the best bet is now returned with the results. Clicking add and publish on the best bet creation / editor screen has the same effect as clicking the publish button on the listing screen.

    exercise creating a best bet 10
  12. Run a search for girls toys. Observe that the best bet is not returned. This is because the trigger for the best bet has been set to the search keyword(s) exactly matches - this means that the best bet will only trigger if the user’s query exactly matches toys. Return to the best bets editor and change the trigger to substring match. Save and publish the best bet and rerun the search for girls toys. Observe that the best bet is returned. This is because the trigger toys is a substring of (or contained within) the user’s query, girls toys.

    exercise creating a best bet 11
Extended exercises: best bets
  1. Experiment creating some additional best bets with different trigger types and ensure that the trigger matching makes sense.

  2. Add some HTML formatting to a best bet so that the thumbnail image of the inventor is displayed beside the best bet. Hint: you may need to use the <#noescape> Freemarker function. 

  3. Configure an service name and thumbnail for the silent films default profile. <<<

6. Synonyms

A synonym by definition is any word that has the same or nearly the same meaning as another in the same language (e.g. lawyer, attorney, solicitor). When compiled together in a database or system of these terms, the result is a thesaurus. Funnelback supports user-defined synonyms that are configured in a similar manner to best bets.

Funnelback uses the defined synonyms to expand or modify the user’s query terms behind the scenes. This allows an administrator to use synonyms for additional query modification beyond the thesaurus-like definition of a synonym.

Synonyms in Funnelback can be used to:

  • expand a term into a set of equivalent terms. E.g. when somebody includes the word lawyer somewhere in a query also search for attorney or solicitor.

  • expand acronyms. E.g. if query includes the term moj also search for ministry of justice.

  • map user language to internal language or non-technical language to equivalent technical terms. Users often don’t know the exact technical words to use and this can prevent them from finding what they are looking for. E.g. map bird flu to H1N1.

  • auto-correct known misspellings. E.g. if a query includes the word qinwa automatically replace this with quinoa. Funnelback does include a spelling suggestion system, but synonyms can enhance the user experience by fixing a misspelling without a user needing to click on an extra did you mean link.

Exercise 17: Synonyms

This exercise shows how you can use synonyms to alter the words entered by a user when they run a search.

  1. Log in to the administration interface and select the inventors collection. Open the synonyms editor by selecting customise synonyms from the customise tab, or switching to the marketing dashboard and selecting synonyms from the left hand menu or by clicking on the synonyms tile.

  2. The synonyms listing screen loads and is very similar to the screen used for listing best bets. Create a new synonym by clicking the add new button.

    exercise synonyms 01
  3. The synonyms editor screen appears allowing the quick entry of multiple synonyms. Create synonyms rules to equate the words nappy and diaper. This requires the creation of three rules that expand each of the words into a search for any of the two words. Add a rule with the following:

    • When these keywords are submitted: nappy

    • Transform them to: [nappy diaper]

    • Apply the transformation if: all words must be present in the search keyword(s), in any order

    exercise synonyms 02

    The first column contains the trigger term (nappy) and is compared with the search query entered by the user. If a match is found (as per the match type in the third column) then the term is transformed to the value in the second column. The square brackets indicate that the terms should be ORed together.

    With this in mind the synonym translates as:

    If the word nappy appears anywhere within the user’s query then search for nappy OR diaper.

    To ensure the words are equated add a second synonym that provides the same expansion, but when diaper is the trigger word.

  4. Add another synonym with the following details:

    • When these keywords are submitted: "white out"

    • Transform them to: ["liquid paper" "white out"]

    • Apply the transformation if: the search keyword(s) exactly matches

    The quotes tell Funnelback to treat the phrase as a single word for the purposes of the expansion.

    With this in mind the synonym translates as:

    If the query matches the phrase white out exactly then search for "liquid paper" OR "white out"

  5. Press the add button. The synonyms screen loads showing the defined synonyms:

    exercise synonyms 03
  6. Test the synonym by searching for nappy from the search box at the top of the marketing dashboard screen, ensuring the preview option is selected from the drop down menu. Observe that the search results include items for diaper. If the word nappy appeared anywhere in the search results then it would be highlighted along with the word diaper.

  7. Publish all the synonyms by clicking the publish all button.

Extended exercises: synonyms
  1. Create a synonym to auto-correct the spelling for barby. Add a rule with the following:

    • When these keywords are submitted: barby

    • Transform them to: barbie

    • Apply the transformation if: all words must be present in the search keyword(s), in any order

  2. Rewrite the white out synonym rule to perform the same synonym expansion rule using a regular expression. You will need to delete the existing rule added in the previous exercise before testing. Hint: the regular expression to use is (?i)\bwhite out\b|\bliquid paper\b.

7. Curator

Curator allows an administrator to define rules and actions that are applied to a query. Each curator rule sets consist of one or more triggers, and one or more actions to perform.

7.1. Curator manage screen

The curator management screen allows an administrator to create, edit, clone, publish and unpublish curator rules.

curator manage screen 01

7.2. Curator triggers

A curator trigger is a set of conditions that when satisfied result in the curator rule running.

curator triggers 01

The curator trigger can be made up of a number of different trigger conditions that are combined to form the overall curator trigger.

Each of the curator trigger conditions consist of a trigger type and any additional fields that are required for the type.

Trigger conditions are collected into trigger groups. Each trigger group contains one or more trigger conditions

curator triggers 02

7.2.1. Trigger types

Curator supports a selection of different trigger types that are used for each condition that makes up a trigger group. Additional fields are required for each trigger and vary depending on the chosen trigger type. Each trigger has a positive and negative form (indicated below in the parentheses).

  • Facet selection: trigger if a specified facet is selected (or not selected)

  • Country of origin: trigger if a search originates (or does not originate) from a specific set of countries. Country of origin is determined from a reverse IP address lookup on the user’s IP address.

  • Date range: trigger if the search is made within (or outside of) a specific date period.

  • Keyword: trigger if the search matches (or does not match) specified keywords. The keywords can be matched to the search as an exact match, substring match, regular expression match or if the search contains all the keywords.

  • URL parameters: trigger if the search URL contains (or does not contain) specific parameter/value combinations

  • Segment/attribute: trigger if the user belongs to (or does not belong to) an industry segment of attribute derived from the user’s IP address.

7.3. Curator actions

curator actions 01

Each curator rule once triggered can execute one or more actions chosen from the following action types:

  • Add to, replace or transform search keywords: modifies the user’s query to add, replace or transform terms within the query. Can be used to provide similar behaviour to synonyms but conditionally triggered.

  • Display a simple message: allows a simple informational message to be returned along with the search results.

  • Display an advert: allows an item equivalent to a best bet to be returned along with the search results

  • Promote results: promotes specific URLs to the top of the set of search results

  • Remove results: removes specific URLs from the set of search results

Exercise 18: Create a simple message curator rule

In this exercise a curator rule will be created to add a simple factual message when a specific keyword is entered.

  1. Log in to the marketing dashboard, select the inventors collection then open the curator manager by clicking on curator in the left hand menu, or clicking on the curator tile.

  2. Add a new curator rule by clicking the add new button

    exercise create a simple message curator rule 01
  3. The curator rule editor loads. Define a name for the curator rule. The rule name needs to be unique and is used to identify the rule in the curator manager. Observe that the title updates as the rule name is entered.

    • Rule name: Inventors day - Germany

    exercise create a simple message curator rule 02
  4. Create a trigger for the curator rule. The trigger defines the conditions that will cause the rule to run. A rule that will be triggered whenever someone searches for anything including the word lamarr will be defined. Add a trigger group for the curator rule by clicking the add new button. This creates a new trigger group and populates it with a blank rule.

    exercise create a simple message curator rule 03
  5. Choose the trigger type by clicking on the dropdown menu. This displays all the trigger options that are available when configuring curator. For this rule we want the default value of search keyword(s) match all the terms.

    exercise create a simple message curator rule 04
  6. Define the additional values required for the trigger type. Enter lamarr into one of the term fields. Remove the other empty term field by clicking the adjacent - button.

    exercise create a simple message curator rule 05
  7. This completes the definition for the trigger. Add an action by either clicking on the then do these actions…​ tab then clicking the add action button, or by clicking the add actions button on the trigger screen.

    exercise create a simple message curator rule 06
  8. Choose an appropriate action type from the action type dropdown menu. Choose:

    • Action: display a simple message

    Observe that the list of fields updates when the action is changed. The fields for defining an action are dependent on the type of action chosen.

    exercise create a simple message curator rule 07
  9. Define the additional action fields. Enter the message into the message box.

    • Message: Did you know that Hedy Lamarr’s birthday, November 9, is Inventors' Day in the German-speaking countries Germany, Austria and Switzerland?

    HTML code can be input into this field and observe that a preview is displayed as the message is input. The preview gives a rough idea of how the message may look, but the actual look and feel in the search results page will be governed by the CSS style sheets that the website designer applies.

    exercise create a simple message curator rule 08
  10. This completes the definition of the action for this curator rule. Save the rule by clicking the green add button. The curator manager reloads displaying the curator rules that are defined for the service.

    exercise create a simple message curator rule 09
  11. The curator rule is now saved but unpublished. This means that it can be previewed using the search box at the top of the marketing dashboard, by running a search and ensuring that preview is selected. Test the rule by running a search for lamarr.

    exercise create a simple message curator rule 10
  12. Observe that the search results for lamarr are displayed and that the message that was just configured is displaying above the search results.

    exercise create a simple message curator rule 11
  13. Return to the curator manager and edit the curator rule by clicking on the rule name. The editor reopens allowing modification of the rule.

    exercise create a simple message curator rule 12
  14. Add a second action. Click on the add actions button, or select the then do these actions…​ tab then click the add another action button. Observe that an empty action is added below the first action.

    exercise create a simple message curator rule 13
  15. Change the action type of the new action to add terms to the search keywords, and enter technology into the terms field. Save the action then re-run the search for lamarr.

    exercise create a simple message curator rule 14
  16. Observe that the order of results returned changes, and that the word technology is highlighted in the result summaries. Also observe that the message is still being displayed. This demonstrates a curator rule that has two actions for a single trigger.

    exercise create a simple message curator rule 15
  17. You can also combine multiple triggers with ANDs and ORs. For example you may want the Inventor’s Day message to be displayed on the actual date, November 9th.

  18. To do so, return to the curator manager and edit the rule.

  19. You may want the rule to trigger when the query "lamarr" is entered AND the date is November 9th. To do so, click on add a trigger group. This will combine triggers with AND. To combine triggers with OR, use the Combine button.

  20. Select the Search is made within a date range trigger type and select the 9th of November for the current year.

    exercise create a simple message curator rule 16
  21. Save the rule and run the search again. The message should not be displayed anymore. You can edit the rule again, change the date today’s date, save, and confirm the rule is displayed again.

  22. Return to the curator manager and publish the curator rule to make it live.

8. Auto-completion

Funnelback’s auto-completion provides the user with real-time suggestions as query terms are typed into the search box.

THe auto-completion returns one or more columns of suggesions, with each column derived from independent sources.

8.1. Auto-completion datasets

Funnelback supports auto-completion returned from one or more sources (auto-completion datasets), usually formatted as separate columns in the auto-completion drop down menu.

auto completion datasets 01

The datasets supported are:

  • Up to 1 simple auto-completion dataset: simple auto-completion is quick and easy to set up and returns suggestions that are words and phrases found within the content.

  • Zero or more structured auto-completion datasets: Structured auto-completion provides advanced auto-completion based on structured CSV data. The suggestions can be grouped by type and provide rich content (such as thumbnail images and other data) in the suggestion. Clicking on a suggestion can run a query, take the user directly to a resource or run some JavaScript.

  • Up to 1 recent searches dataset: Recent searches auto-completion is derived from the user’s search history. This is available when sessions are enabled on a user’s search.

The example above shows auto-completion from three independent datasets. The suggestions column is simple auto-completion from words and phrases present in the complete index being searched. The people and courses columns are both structured auto-completion based on structured CSV data.

The general approach for configuring the datasets is to have the simple auto-completion configured on the profile that is queried by the user (usually _default), and additional profiles created for each CSV dataset.

For this approach to work the Funnelback collection should be configured to generate auto-completion using spelling suggestions and collection CSV files.

By default Funnelback will generate simple auto-completion. If you are using the default Funnelback template the simple auto-completion should automatically work after you have updated your collection.

Exercise 19: Attach auto-completion to a Funnelback template

This exercise outlines the bits of code you need to add to a template in order to get auto-completion working with the Concierge plugin. If you’re using the default Funnelback template the code will already be there.

  1. Log in to the administration interface by visiting https://training-admin.clients.funnelback.com/search/admin/

  2. Switch to the inventors collection.

  3. Edit the default template by selecting edit results template from the customise tab.

  4. Locate the search box (hint: it is an input with a name=query) and ensure it has a unique ID parameter. The search box should already have id=query set so no change is required to the search box for this page in order to add the auto-completion. When working with your own search boxes you will need to ensure you have a unique id assigned. This provides a unique identifier can be used to attach the auto-completion.

    exercise attach auto completion to a funnelback template 01
  5. Load the required JavaScript libraries. Ensure that the jQuery, Typeahead, Handlebars and the funnelback.autocompletion-2.6.0.js JavaScript libraries are loaded from the template source. Add the following code to the bottom of the template, before the </body> tag. The ${GlobalResourcesPrefix} variable is expanded into the base path on the Funnelback server so that the full URL will reference the correct file. The <#if> statement just checks to see if auto-completion is enabled for the search before importing the JavaScript libraries. Note: these files should be loaded in the order below.

    <script src="${GlobalResourcesPrefix}thirdparty/jquery-3.3.1/jquery.min.js"></script>
    <#-- Standard typeahead and handlebars libraries - required for auto-completion -->
    <#if question.currentProfileConfig.get('auto-completion') == 'enabled'>
      <script src="${GlobalResourcesPrefix}js/typeahead.bundle-0.11.1.min.js"></script>
      <script src="${GlobalResourcesPrefix}thirdparty/handlebars-4.1/handlebars.min.js"></script>
      <script src="${GlobalResourcesPrefix}js/funnelback.autocompletion-2.6.0.js"></script>
    </#if>
  6. Configure auto-completion options for the funnelback.autocompletion-2.6.0.js by adding the following block of JavaScript to the bottom of the file, immediately after the funnelback.autocompletion-2.6.0.js is loaded. This block of code attaches the autocompletion function to the element with the ID of query and configures a list of parameters that will be passed to the auto-completion web service by the JavaScript function.

    <#if question.currentProfileConfig.get('auto-completion') == 'enabled'>
      <script>
        jQuery(document).ready(function() {
          jQuery('#query').autocompletion({
            datasets: {
              <#if question.currentProfileConfig.get('auto-completion.standard.enabled')?boolean>
              organic: {
                collection: '${question.collection.id}',
                profile : '${question.profile}',
                program: '<@s.cfg>auto-completion.program</@s.cfg>',
                format: '<@s.cfg>auto-completion.format</@s.cfg>',
                alpha: '<@s.cfg>auto-completion.alpha</@s.cfg>',
                show: '<@s.cfg>auto-completion.show</@s.cfg>',
                sort: '<@s.cfg>auto-completion.sort</@s.cfg>',
                group: true
              },
              </#if>
            },
            length: <@s.cfg>auto-completion.length</@s.cfg>
          });
        });
      </script>
    </#if>

    If you wish to attach auto-completion to a search box with an ID of searchbox then you would replace

    jQuery('#query').autocompletion({

    with

    jQuery('#searchbox').autocompletion({

    You can also define multiple code blocks similar to the one above, but attaching to different IDs if you wish to have auto-completion bind to more than one search box. Each code block can be configured with different auto-completion sources and configuration parameters.

  7. Add CSS styling to handle the auto-completion suggestions. Add the following CSS import to the <head> section of the template.

      <#if question.currentProfileConfig.get('auto-completion') == 'enabled'>
      <link rel="stylesheet" type="text/css" href="${GlobalResourcesPrefix}css/funnelback.autocompletion-2.6.0.css" />
      </#if>
  8. Test the auto-completion by previewing the search results page and start typing a query into the search box. Suggestions should be returned as you type.

    exercise attach auto completion to a funnelback template 02

The above process can be used to attach Funnelback driven auto-completion to any search box on your website - the search box does not need to be part of a Funnelback search results page.

The HTML code needs to be cleaned when attaching to a search box that is not part of a Funnelback template (.ftl) file - any Freemarker macros or variables in the above examples will need to be removed for these non-Funnelback template pages. This means that full paths to the funnelback.autocompletion-2.6.0.js will be required and the configuration options that are auto-filled in the JavaScript configuration code (see above exercise) will need to be expanded to suitable values.

The simplest way to achieve this may be to view the HTML source of a search results page that includes the configured auto-completion and copy this into your web page, then fix any URLs that need to be made into absolute URLs.

e.g. inspect the source code for the page in the previous example and observe the auto-completion JavaScript block.

attach auto completion to any search box 01

Placing the highlighted code, plus the JavaScript imports (jQuery, Typeahead etc.) and the auto-completion stylesheet include into your own web page or site template allows you to attach auto-completion to your website’s search box.

8.3. Auto-completion Javascript plugin compatibility

The funnelback.autocompletion-2.6.0.js library that ships with Funnelback has some Javascript dependencies.

The funnelback.autocompletion-2.6.0.js code is only tested the bundled versions of jQuery, Typeahead and Handlebars. Although it has been found to be compatible with other versions, correct functioning is not guaranteed and the funnelback.autocompletion-2.6.0.js file should be treated as a sample implementation of integrating with the auto-completion web service. It should also be noted that if it is integrated with a website that also uses any of these dependencies then the website should be updated to use the same versions of as Funnelback, or some changes might be required to the funnelback.autocompletion-2.6.0.js code that is imported into existing website designs.

Funnelback’s auto-completion web service which returns the suggestions as a JSON packet is not tied to any particular technologies or versions of JQuery.

8.4. Auto-completion configuration

There are a numerous options available when configuring auto-completion.

A number of settings control the generation of auto-completion indexes and how they are interpreted by the auto-completion web service.

The other settings control how the Javascript auto-completion plugin interacts with the web service and displays the suggestions.

These options configure parameters that are sent to Funnelback’s auto-completion web service controlling behaviour such as the number of suggestions to request, the number of keystrokes that are entered before the first request is made and how to sort the suggestions. Options also define how the auto-completion is displayed and other aspects of how the auto-completion behaves.

These options can be adjusted via the administration interface and configure the integration of funnelback.autocompletion-2.6.0.js with the auto-completion web service.

The auto-completion options set via the administration interface will only have an effect if you use the included JavaScript (funnelback.autocompletion-2.6.0.js) or if your implementation is designed to read the same options from Funnelback’s configuration and will only affect pages generated by Funnelback. A site search box that has been configured to display auto-completion has the values hardcoded in the JavaScript.
Exercise 20: Configure simple auto-completion
  1. Access the administration interface and switch to the inventors collection.

  2. Select edit profile configuration from the administer tab.

    exercise configure simple auto completion 01
  3. Configure a simple auto-completion dataset. Check to see if a auto-completion.source parameter is set for the service. Click add new and look up the auto-completion.source. For simple auto-completion this needs to be set to a value of internal. Save and then publish the setting.

    exercise configure simple auto completion 02

    Observe that the default value for auto-completion.source is internal. This means you don’t need to set the auto-completion.source but you can if you want to. For this example set the value as it will mean that you can see that this has been set when you view the profile configurations. This is all that is required to configure a simple auto-completion source. If you change the value of auto-completion.source you will need to rebuild the search index.

  4. We we now make some basic changes to how the auto-completion plugin works. The auto-completion plugin is a Javascript program that is designed to interact with Funnelback’s auto-completion web service - sending through the user’s keystrokes as they are typed and formatting the set of returned suggestions.

    Set auto-completion to occur when only one character is entered by the user. Adjust the minimum number of keystrokes before auto-completion is triggered. Set the following configuration key: auto-completion.length to a value of 1. Don’t forget that you need to publish the setting for it to take effect on the live profile.

    exercise configure simple auto completion 03
  5. Preview the search and test the effect of this change. Auto-completion suggestions should now start to appear after a single character is entered into the search box.

    exercise configure simple auto completion 04
  6. Inspect the page source and observe the JavaScript configuration block at the end of the HTML file has been populated with the values from the administration interface. The other values within the configuration are populated from the default configuration values.

    If you are integrating auto-completion with your site’s search box use this generated code as a guide for how to set the options in your page code.
    <script>
      jQuery(document).ready(function() {
        jQuery('#query').autocompletion({
          datasets: {
            organic: {
              collection: 'inventors',
              profile : '_default_preview',
              program: '../s/suggest.json',
              format: 'extended',
              alpha: '0.5',
              show: '10',
              sort: '0',
              group: true
            },
          },
          length: 1
        });
      });
    </script>

8.5. Structured auto-completion

The search suggestions displayed in simple auto-completion are words found in the indexed content. This improves a user’s search experience by reducing the chance of running a search that doesn’t return any results. These suggestions are generated automatically as part of the update and are a very powerful enhancement that can be added to the search.

Structured auto-completion is an extension of the functionality provided by simple auto-completion. Structured auto-completion generates suggestions based off auto-completions defined in CSV data. The suggestions can be grouped by type and provide rich content (such as thumbnail images and other data) in the suggestion. The action performed when clicking on a suggestion is configurable and can run a query, take the use directly to a resource or run some JavaScript.

The CSV file can be edited directly from within Funnelback or uploaded via the file manager.

The CSV file can also be generated from an external source such as a database, or even the metadata within the Funnelback index and imported into Funnelback as part of an update using workflow - however this is beyond the scope of FUNL201 and is covered in FUNL203.

The CSV file format used by auto-completion is detailed in the Funnelback documentation

The CSV file contains 8 columns:

  1. Trigger: This is the word, or set of words that will trigger the suggestion to display. The keywords that are input by the user are left matched against this trigger.

  2. Weight: This is the weighting to apply to the suggestion. Range is 0 - 999. This weight is used to determine the order that suggestions will display when default sorted. Choose a high value (eg. 900) as your starting point and work from there - for most applications having the same weight of 900 for all the rich suggestions works well. The weighting is returned in the JSON (when you view the auto-completion web service response - see earlier section from the organic auto-completion exercise).

  3. Display content: This is what is displayed to the user - it can be plain text (like for the organic suggestions, a html fragment (as nested within a DIV), a JavaScript statement to execute or JSON data.

  4. Display type: This field indicates the type of the display content used by the display content field.

  5. Category group: This field can contain a string that is used to group the suggestions.

  6. Category type: This field is currently unused and is left blank.

  7. Action: This field contains either a keyword string, a URL or a JavaScript callback function. A keyword string will be run as a search (as though the user input the string into the search box, a URL will take a user directly to the URL and the JavaScript callback will be executed.

  8. Action type: This field indicates the type of action above. U (for URLs) is the most common choice for rich auto-completion.

Exercise 21: Structured auto-completion
  1. Load the administration interface and switch to the inventors collection

  2. Create a profile for each CSV dataset. From the administer tab select manage profiles and create a new profile called wikipedia. We will add custom structured completions snippets coming from Wikipedia data.

  3. Return to the main admin page by either clicking the Funnelback logo or by clicking the cancel button and switch to the wikipedia profile.

  4. All profiles will inherit the organic completions from the collection content. To ensure we only get CSV completion in this profile, we need to configure the source data for completions and limit it to the CSV data.

    exercise structured auto completion 01
  5. Configure a structured auto-completion dataset. Select edit profile configuration from the administer tab and add an auto-completion.source. Set the value to csv then save and publish the setting. This configures Funnelback to generate the auto-completion for the wikipedia profile using a CSV file named auto-completion.csv that is configured as part of the wikipedia profile.

    exercise structured auto completion 02
  6. Open the file manager (select browse collection configuration settings from the administer tab) and create a new auto-completion.csv file by selecting the auto-completion.csv option from the create menu located within the inventors / Profile: wikipedia (preview) section then clicking create.

    exercise structured auto completion 03
  7. Paste the following CSV into the editor window and then save the file.

    mary anderson,900,{\"name\":\"Mary Anderson\"\, \"yob\":\"1866\"\, \"yod\":\"1953\"\, \"thumb\":\"http://training-search.clients.funnelback.com/training/training-data/inventors-resources/mary_anderson.jpg\"\, \"inventions\":\"windshield wipers (1903)\"},J,Inventors,,https://en.wikipedia.org/wiki/Mary_Anderson_(inventor),U
    anderson,890,{\"name\":\"Mary Anderson\"\, \"yob\":\"1866\"\, \"yod\":\"1953\"\, \"thumb\":\"http://training-search.clients.funnelback.com/training/training-data/inventors-resources/mary_anderson.jpg\"\, \"inventions\":\"windshield wipers (1903)\"},J,Inventors,,https://en.wikipedia.org/wiki/Mary_Anderson_(inventor),U
    windshield wipers,900,{\"name\":\"Mary Anderson\"\, yob:\"1866\"\, yod:\"1953\"\, \"thumb\":\"http://training-search.clients.funnelback.com/training/training-data/inventors-resources/mary_anderson.jpg\"\, \"inventions\":\"windshield wipers (1903)\"},J,Inventors,,https://en.wikipedia.org/wiki/Mary_Anderson_(inventor),U
    wipers,890,{\"name\":\"Mary Anderson\"\, \"yob\":\"1866\"\, \"yod\":\"1953\"\, \"thumb\":\"http://training-search.clients.funnelback.com/training/training-data/inventors-resources/mary_anderson.jpg\"\, \"inventions\":\"windshield wipers (1903)\"},J,Inventors,,https://en.wikipedia.org/wiki/Mary_Anderson_(inventor),U
    barbara askins,900,{\"name\":\"Barbara Askins\"\, \"yob\":\"1939\"\, \"yod\":\"\"\, \"thumb\":\"http://training-search.clients.funnelback.com/training/training-data/inventors-resources/barbara_askins.jpg\"\, \"inventions\":\"new method for developing film (1978)\"},J,Inventors,,https://en.wikipedia.org/wiki/Barbara_Askins,U
    askins,890,{\"name\":\"Barbara Askins\"\, \"yob\":\"1939\"\, \"yod\":\"\"\, \"thumb\":\"http://training-search.clients.funnelback.com/training/training-data/inventors-resources/barbara_askins.jpg\"\, \"inventions\":\"new method for developing film (1978)\"},J,Inventors,,https://en.wikipedia.org/wiki/Barbara_Askins,U
    margaret knight,900,{\"name\":\"Margaret Knight\"\, \"yob\":\"1838\"\, \"yod\":\"1914\"\, \"thumb\":\"http://training-search.clients.funnelback.com/training/training-data/inventors-resources/margaret_knight.jpg\"\, \"inventions\":\"flat bottomed paper bag (1978)\"},J,Inventors,,https://en.wikipedia.org/wiki/Margaret_E._Knight,U
    knight,890,{\"name\":\"Margaret Knight\"\, \"yob\":\"1838\"\, \"yod\":\"1914\"\, \"thumb\":\"http://training-search.clients.funnelback.com/training/training-data/inventors-resources/margaret_knight.jpg\"\, \"inventions\":\"flat bottomed paper bag (1978)\"},J,Inventors,,https://en.wikipedia.org/wiki/Margaret_E._Knight,U
    ruth handler,900,{\"name\":\"Margaret Knight\"\, \"yob\":\"1916\"\, \"yod\":\"2002\"\, \"thumb\":\"http://training-search.clients.funnelback.com/training/training-data/inventors-resources/ruth_handler.jpg\"\, \"inventions\":\"Barbie doll (1959)\"},J,Inventors,,https://en.wikipedia.org/wiki/Ruth_Handler,U
    handler,890,{\"name\":\"Margaret Knight\"\,\"yob\":\"1916\"\, \"yod\":\"2002\"\, \"thumb\":\"http://training-search.clients.funnelback.com/training/training-data/inventors-resources/ruth_handler.jpg\"\, \"inventions\":\"Barbie doll (1959)\"},J,Inventors,,https://en.wikipedia.org/wiki/Ruth_Handler,U
    baby,900,"Disposable diaper (1949)",T,Inventions,,baby,Q
    baby,900,"Snugli baby carrier (1969)",T,Inventions,,baby,Q
    windshield wipers,900,"Windshield wipers (1903)",T,Inventions,,windshield wipers,Q
    car,900,"Windshield wipers (1903)",T,Inventions,,windshield wipers,Q

    This creates eight different suggestions, some of which have been duplicated several times with different triggers. The suggestions are grouped into two types - inventors and inventions. In this CSV file the inventor type suggestions are JSON type suggestions that have Wikipedia pages as the target when you click on the suggestion. The invention type suggestions are text type suggestions that result in a query being run when you click on the suggestion.

    Auto-completion is only generated for live profiles so it is important that you do not forget to publish the CSV file.
  8. Locate the CSV file in the file manager and publish the file.

  9. Rebuild the index (select start advanced update from the update tab, then select reindex live view and click update) to generate the auto-completion.

  10. View the search results and perform a search for margaret observing the suggestions have not changed. This is because the JavaScript needs to be reconfigured to request the additional auto-completion dataset.

  11. Update the JavaScript code as shown below (hint: you will need to edit your template on the _default profile):

    1. Add a dataset called wikipedia to the JavaScript auto-completion configuration. Note: make sure the case for the profile parameter matches the profile you created (profile: 'wikipedia'). If you named it Wikipedia then make sure you use profile: 'Wikipedia' when configuring concierge.

    2. add a header to the organic suggestions

    3. configure the auto-completion to display in columns

    4. configure the auto-completion program at the global level.

      <#if question.currentProfileConfig.get('auto-completion') == 'enabled'>
        <script>
          jQuery(document).ready(function() {
            jQuery('#query').autocompletion({
              program: '<@s.cfg>auto-completion.program</@s.cfg>',
              horizontal: true,
              datasets: {
                <#if question.currentProfileConfig.get('auto-completion.standard.enabled')?boolean>
                organic: {
                  collection: '${question.collection.id}',
                  profile : '${question.profile}',
                  format: '<@s.cfg>auto-completion.format</@s.cfg>',
                  alpha: '<@s.cfg>auto-completion.alpha</@s.cfg>',
                  show: '<@s.cfg>auto-completion.show</@s.cfg>',
                  sort: '<@s.cfg>auto-completion.sort</@s.cfg>',
                  template: {
                    header: '<h4>Suggestions</h4>'
                  },
                  group: true
                },
                wikipedia: {
                  collection: '${question.collection.id}',
                  profile : 'wikipedia',
                  show: '3',
                  template: {
                    header: '<h4>Wikipedia</h4>',
                    suggestion: '<div>{{#if label.name}}{{#if label.thumb}}<img src="{{label.thumb}}" style="height:40px; float:left; clear:both; padding:1px; border:1px solid #333; margin-right:2px;"/>{{/if}}{{label.name}} ({{label.yob}} - {{label.yod}})<br/><span style="font-size:80%;"><em>Invented: {{label.inventions}}{{else}}{{label}}{{/if}}</em></span></div>',
                    group: '<h5>{{label}}</h5>'
                  },
                  group: true
                },
                </#if>
              },
              length: <@s.cfg>auto-completion.length</@s.cfg>
            });
          });
        </script>
      </#if>
  12. Start to search for baby and observe the two columns of suggestions and how they change as you type. If this doesn’t work, double check that you have published the profile configuration options and auto-completion.csv files (and re-index if needed).

    exercise structured auto completion 04
  13. Click on the Barbara Askins suggestion and observe that you are taken directly to the Wikipedia page that you configured.

    exercise structured auto completion 05
  14. Repeat the search and this time click on the disposable diaper suggestion. Observe that a search for baby is run - this aligns with the action that was configured in the CSV.

    exercise structured auto completion 06

8.6. Auto-completion web service

Many of the auto-completion configuration options available via the administration interface only control the call made to the JSON endpoint producing suggestions.

The funnelback-completion-2.6.0.js JavaScript is responsible for requesting auto-completion suggestions based on what has been entered into the search box, submitting the request after each keystroke and transforming the JSON response from the web service into the HTML presented to the end user.

As a front-end developer might wish to use other JavaScript frameworks or libraries to interact with the web service - your code will need to form these calls on the user’s behalf and transform and insert the response into page DOM.

Any implementation that interacts with the auto-completion web service will need to handle the construction of the suggest.json request URL and also the processing and conversion of the JSON response into the final HTML that’s returned to the end user.

Exercise 22: Auto-completion web service

This exercise examines the raw request that is submitted to the auto-completion web service and JSON response.

  1. Load the inventors search interface in your browser by using the preview function from the administration interface, or by directly visiting: http://training-search.clients.funnelback.com/s/search.html?collection=inventors

  2. Open up your browser debugger and ensure network traffic is being inspected.

  3. Start typing a query into the search box and observe that a call to suggest.json is made to each configured auto-completion dataset for each keystroke.

    exercise auto completion web service 01
  4. Each /s/suggest.json call is a request to the auto-completion web service. Observe that each call specifies the options set in the JavaScript and the only difference between subsequent calls is the value of the partial query which reflects the current partially entered query of the search box.

    http://training-search.clients.funnelback.com/s/suggest.json?collection=inventors&fmt=json%2B%2B&alpha=0.5&profile=wikipedia&show=3&partial_query=b
    http://training-search.clients.funnelback.com/s/suggest.json?collection=inventors&fmt=json%2B%2B&alpha=0.5&profile=wikipedia&show=3&partial_query=ba
  5. Open one of the URLs in your browser and inspect the JSON that is returned. Compare this with what was returned in the list of suggestions displayed in the search box when you typed.

    This is an example of simple auto-completion (note: the JSON has been truncated in this example):

    {
    key: "barbara",
    disp: "barbara",
    disp_t: "T",
    wt: "4.899",
    cat: "",
    cat_t: "",
    action: "",
    action_t: "S"
    },
    {
    key: "bank",
    disp: "bank",
    disp_t: "T",
    wt: "1.414",
    cat: "",
    cat_t: "",
    action: "",
    action_t: "S"
    },
    {
    key: "barbie",
    disp: "barbie",
    disp_t: "T",
    wt: "1.414",
    cat: "",
    cat_t: "",
    action: "",
    action_t: "S"
    },
    {
    key: "baby",
    disp: "baby",
    disp_t: "T",
    wt: "1.414",
    cat: "",
    cat_t: "",
    action: "",
    action_t: "S"
    }
    ]

    And this is an example of the structured auto-completion returned:

    [
    {
    key: "barbara askins",
    disp: {
    name: "Barbara Askins",
    yob: "1939",
    yod: "",
    thumb: "http://training-search.clients.funnelback.com/training/training-data/inventors-resources/barbara_askins.jpg",
    inventions: "new method for developing film (1978)"
    },
    disp_t: "J",
    wt: "900",
    cat: "Inventors",
    cat_t: "",
    action: "https://en.wikipedia.org/wiki/Barbara_Askins",
    action_t: "U"
    },
    {
    key: "baby",
    disp: "Disposable diaper (1949)",
    disp_t: "T",
    wt: "900",
    cat: "Inventions",
    cat_t: "",
    action: "baby",
    action_t: "Q"
    },
    {
    key: "baby",
    disp: "Snugli baby carrier (1969)",
    disp_t: "T",
    wt: "900",
    cat: "Inventions",
    cat_t: "",
    action: "baby",
    action_t: "Q"
    }
    ]
Extended exercises: Auto-completion

The following resources will assist in the extended exercises and also provide additional information and options for configuring the auto-completion plugin.

  1. In the browser debugger observe that separate calls to suggest.json are made for to each dataset each time auto-completion is triggered. Inspect the JSON for each dataset and compare to the menus that are rendered in the browser, and also the template defined in the JavaScript configuration

  2. Configure the plugin to return 5 organic suggestions, and 1 wikipedia suggestion. (hint: configure the auto-completion.show option)

  3. Configure the plugin to return the different datasets stacked (hint: see the horizontal JavaScript option)

  4. Configure the plugin to only return suggestions after 3 keystrokes (hint: configure the auto-completion.length option)

  5. Configure the plugin to display some default suggestions before the user types (hint: see the defaultCall JavaScript option)

  6. Configure no suggestion templates for each dataset (hint: see the notFound template)

  7. Attach auto-completion to a non-Funnelback search form.

Search queries are often short, vague and ambiguous. Contextual navigation helps a user to narrow their search and extract additional context from a user’s intent by suggesting types and topics related to their current search query. Contextual navigation is built dynamically by Funnelback by analysing patterns in the text contained within the search results and suggesting a number of related searches.

Clicking on a suggestion will always return search results.

Contextual navigation is enabled by default and returns related searches grouped by type, topic and site.

Each related search suggestion is actually a proximity search.

A proximity search is where the search terms must appear near to each other in the content. Proximity searches are specified in the Funnelback query language using backtick (`) operators.

The nearness is controlled by the prox query processor option and defaults to 15 words.

e.g. a search for `silent comedy` will return results containing silent and comedy, as long as silent and comedy are not separated by more than 15 words.

9.1. Contextual navigation basic configuration options

The following options change the basic behaviour of contextual navigation:

contextual-navigation.enabled

This setting enables display of the contextual navigation output for the selected collection. Note that contextual navigation will not appear on the search results page without a valid template.

contextual-navigation.categorise_clusters

This setting groups the suggestions into types and topics based on the query. The categorisation algorithm is not always accurate, but in most cases provides a helpful grouping of the suggestions. If set to false, types and topics are grouped together into topics. The default is true.

contextual-navigation.type.max_clusters

This setting limits the number of type suggestions displayed on the search results page. The default is 15 type suggestions.

contextual-navigation.topic.max_clusters

This setting limits the number of topic suggestions displayed on the search results page. The default is 15 topic suggestions.

contextual-navigation.site.max_clusters

This setting limits the number of site suggestions displayed on the search results page. The default is 8 site suggestions.

contextual-navigation.kill_list

Suggestions containing any of these words or phrases will not be displayed as a related search.

contextual-navigation.cannot_end_with

Suggestions ending in any of these words or phrases will not be displayed as a related search.

contextual-navigation.case_sensitive

Words in this list found in any suggestions will be treated as case sensitive when generating the suggestions and case will be preserved.

contextual-navigation.summary_fields

The summary fields are Funnelback metadata classes that are analysed in addition to the page content for the purpose of generating the suggestions. The default summary fields are description (c) and subject (keyword) metadata. Any other metadata classes that contain descriptive information can be added to this list to further enhance the quality of suggestions.

Exercise 23: Configuring contextual navigation

The default configuration for contextual navigation provides default values that allow you to use the feature without making any changes. The most common change is to change the number of suggestions returned. In this exercise we will show the effect of disabling contextual navigation and then the how to change the number of suggestions that are returned.

  1. Load the administration interface and switch to the silent films collection

  2. Perform a search against the silent-films collection for comedy. The search for the silent films collection can be accessed from: http://training-search.clients.funnelback.com/s/search.html?collection=silent-films

  3. Observe the contextual navigation suggestions (related searches) that appear below the search results.

    exercise configuring contextual navigation 01
  4. Contextual navigation is configured by setting a number of collection-level configuration settings. Select edit collection configuration from the administer tab then select contextual navigation from the left hand menu. Disable contextual navigation by changing the value of the contextual_navigation.enabled setting from true to false:

    exercise configuring contextual navigation 02
  5. Re-run the search for comedy, observing that the contextual navigation suggestions disappear.

  6. Return to the configuration editor screen and re-enable the contextual navigation by reverting the configuration setting. Change the number of type and topic clusters (contextual-navigation.topic.max_clusters and contextual-navigation.type.max_clusters) displayed to a maximum of 2 then save the changes.

  7. Rerun the search for comedy and confirm that no more than two suggestions are appearing under each heading:

    exercise configuring contextual navigation 03

10. Controlling display options

There are numerous display options that can be set, controlling various aspects of the display.

These options can be set as default behaviour for your search, or passed in as parameters when a search is run.

The options below are some of the more commonly used display options:

  • Number of results per page: this is the number of search results to return for each results page. Parameter name is num_ranks. (Default is 10 results: num_ranks=10)

  • Summary mode: controls the type of result summaries returned with each result. Possible modes are:

    • Automatic: Summaries will be returned for each result. Summary source will be determined automatically by Funnelback. This is the default. (SM=auto) note: this is also equivalent to the old SM=both summary mode.

    • Disabled: Summaries are disabled for all search results (set using SM=off). This also prevents metadata from being returned in the data model.

    • Snippet: A short snippet from the start of the document is returned as the result summary. (set using SM=snip)

    • Query biased: A series of short extracts are returned for each result. Each extract is biased to the keywords used for the search and shows the matched term(s) in context to the surrounding words. The number and order of extracts can be controlled using the EORDER and SQE options (set using SM=qb)

    • Metadata only: Return only metadata fields as defined by the SF parameter (set using SM=meta).

  • Metadata fields: Controls the metadata fields that are returned with the search results (as long as the summary mode is not set to off). Set using the SF parameter.

  • Summary length: Controls the size (in number of characters) of the buffer used to store the summaries. Increasing this will result in longer summaries being returned. Set using the SBL parameter.

  • Metadata length: Controls the size of the buffer used to store metadata summaries. This value may need to be increased if metadata values are being cut off. Set using the MBL parameter.

  • Sort mode: Controls the sort order for the search results. Sort can be by relevance/search score (default), title, metadata field, date or URL, in either ascending or descending direction. Set using the sort parameter.

A full list of display options is available in the documentation.

Exercise 24: Dynamically alter the sort using URL parameters

Several sort modes are provided for results - relevance is set as the default, but we can easily experiment with others by adding URL parameters to any search results page.

  1. Perform a search against the silent-films collection for death. The search for the silent-films collection can be accessed from: http://training-search.clients.funnelback.com/s/search.html?collection=silent-films.

    exercise dynamically alter the sort using url parameters 01
  2. Add &sort=title to the URL and observe the same number of results in the set, reordered by title (A-Z).

    exercise dynamically alter the sort using url parameters 02
  3. Edit the URL to sort by URL &sort=url and observe the same number of results in the set, this time reordered by URL (A-Z).

    exercise dynamically alter the sort using url parameters 03

A full list of sort modes is detailed in the Funnelback documentation.

Exercise 25: Change the default sort mode

The default sort mode is to sort by relevance (or search score). The default can be changed by setting a display option.

  1. Load the administration interface and switch to the silent-films collection.

  2. Select edit collection configuration from the administer tab then select interface from the left hand menu.

    exercise change the default sort mode 01
  3. The query processor is the Funnelback program that returns search results from the index based on search keywords. Query processor options are the ranking and display options that are passed to the query processor when it runs the search. Edit the query_processor_options and add -sort=title to the set of options then save the setting. This will set sort by title as the default sort mode. The value (title) can be substituted for any of the other sort modes that are available from the documentation page above.

    exercise change the default sort mode 02
  4. Run a search against the silent-films collection for death. The search for the silent-films collection can be accessed from: http://training-search.clients.funnelback.com/s/search.html?collection=silent-films.

  5. Observe that the sorting is now alphabetic. Experiment by modifying the URL (as in the previous exercise) and observe that the sort mode can be overridden at the time the query runs, but will also be alphabetic when no sort parameter is supplied.

    exercise change the default sort mode 03
  6. Return to the configuration editor and revert the options, removing the -sort=title option to restore relevancy as the default search.

Exercise 26: Create a sort-mode switcher in the template

The previous two exercises have demonstrated the setting of sort both via configuration and dynamically at the time the query is run.

In this example a switch control will be added to the search results listing to enable the user to re-sort the result set.

  1. Load the administration interface and switch to the silent-films collection.

  2. Edit the template (by selecting design results page from the customise tab).

  3. Add the following code snippet above your result set output (e.g. just before the ol id="search-results" (approx line 501). Save and publish once complete:

    <#-- SEARCH AGAIN -->
        <form action="${question.currentProfileConfig.get("ui.modern.search_link")}" method="GET" id="sortform">
            <#-- PRE-FILL PARAMETERS -->
            <input type="hidden" name="collection" value="${question.collection.id}" />
            <input type="hidden" name="form" value="${question.form }" />
            <input type="hidden" name="profile" value="${question.profile}" />
            <input type="hidden" name="query" value="<@s.QueryClean/>" />
            <#-- SORT MODE -->
            <label for="sort">Sort</label>
            <@s.Select id="sort" name="sort" options=["=Relevance ", "date=Date (newest first)", "adate=Date (oldest first)","title=Title"] />
            <input type="submit" value="Go" />
        </form>
  4. Run a search of the silent-films collection for death. http://training-search.clients.funnelback.com/s/search.html?collection=silent-films

  5. Observe that a sort dropdown is returned above the search results. Experiment with the sort mode drop-down observing the change in sort order while maintaining the same set of search keywords.

    exercise create a sort mode switcher in the template 01
Extended exercises: display options
  1. Create a switcher that allows a user to choose the number of results that they wish to see.

  2. Experiment with changing the summary mode setting by editing your URL and observe the effect on the search results.

11. Metadata

11.1. What is metadata?

Metadata is information about a document. Metadata can come from a number of sources - it can be defined in HTML <meta> tags within a web page (such as description metadata), other HTML tags (such as the document’s <title> element), or inferred from other information about an item (such as the size of the document in bytes, or the URL of the document).

In Funnelback metadata is also used for fielded information such as those sourced from XML, JSON, CSV or databases.

11.2. Why use metadata?

Metadata can be used in a number of different ways by Funnelback to enhance your search results and provide users with an enhanced search experience.

In general metadata can be used in the following ways:

  • To provide additional keyword data for the purposes of improving ranking

  • To provide information that can be used for display purposes (such as for custom result summaries)

  • As a way of classifying documents, allowing for enhanced functionality (such as filtering search results by fields of particular values)

  • As additional information that Funnelback can use for other features (such as to generate structured auto-completion).

Metadata can also be incorporated from an external source (such as a database) by producing external metadata. External metadata is covered in a later exercise.

The screenshot below illustrates how metadata can be used to enhance your search results. For this search the metadata values contained within tag metadata are used to produce faceted navigation that allows the search results to be filtered based on the selected tags. A metadata field containing a URL to a thumbnail image is also used to add a thumbnail to each search result that has a defined thumbnail image.

why use metadata 01

The amount of enhancement to the results that can be achieved is really only limited by the metadata you have available and could be used for example to conditionally display things in the search results. For example:

  • If it’s a video then open a Javascript lightbox to play the video when you click on the link.

  • Display a curated description for items where description metadata has been authored.

  • Display some extra fielded information about each item (type, location, position information)

  • Provide a small map for each result that contains geospatial metadata plotting the item on the map and populating an info bubble.

  • Display file type icons next to each binary file search result.

11.3. Metadata classes and mapping

Metadata is read by Funnelback as part of the process of building the search indexes. The source and type of any metadata that should be included needs to be configured before it will be indexed.

The configuration involves identifying the metadata source (e.g. a metadata field name), assigning a type and defining a mapping to a Funnelback metadata class. Once this has been configured, metadata will become available in the search index.

11.3.1. Metadata classes

Metadata classes are used by Funnelback to organise and access metadata. Metadata classes are given a unique identifier and type and one or more metadata sources are mapped to each class.

Metadata classes are used when configuring other Funnelback features that use metadata such as faceted navigation or contextual navigation.

11.3.2. Mapping metadata sources to classes

Metadata can come from a number of different sources, and these sources must be mapped to a metadata class in order for the metadata to be searchable by Funnelback.

Funnelback currently supports the following sources of metadata:

  • HTML meta tags such as <meta name="dc.title" content="This is the title">.

  • HTML tags such as <h1> or <title>. Note: CSS-class style selectors are not supported.

  • HTTP headers such as Content-Type or X-Funnelback-Custom-Header.

  • XML X-Paths such as /books/book/title or //author. Note: not all X-Paths are supported.

A metadata source can only be mapped to a single Funnelback metadata class, however multiple metadata fields can be mapped to the same Funnelback metadata class. This is useful if there are different fields that contain the same functional information (e.g. dc.description, dcterms.description, description).

Metadata sources are grouped into two types:

  • HTML type sources (HTML <meta> tags, HTML tags and HTTP headers) are only considered when indexing web pages and PDF documents.

  • XML type sources are only considered when indexing an XML document.

Funnelback uses XML to represent most non-web collection types such as database and LDAP directory records, social media content, JSON and CSV files. The XML source configuration is required when configuring metadata for these.

11.3.3. Metadata class types

Funnelback supports five types of metadata classes:

  • Text: The content of this class is a string of text.

  • Geospatial x/y coordinate: The content of this field is a decimal latlong value in the following format: geo-x;geo-y (e.g. 2.5233;-0.95) This type should only be used if there is a need to perform a geospatial search (e.g. This point is within X km of another point). If the geospatial coordinate is only required for plotting items on a map then a text type field is sufficient.

  • Number: The content of this field is a numeric value. Funnelback will interpret this as a number. This type should only be used if there is a need to use numeric operators when performing a search (e.g. X > 2050). If the field is only required for display within the search results a text type metadata class is sufficient.

  • Document permissions: The content of this field is a security lock string defining the document permissions. This type should only be used when working with an enterprise collection that includes document level security.

  • Date: A single metadata class supports a date, which is used as the document’s date for the purpose of relevance and date sorting. Additional dates for the purpose of display can be indexed as either a text or number type metadata class.

11.3.4. Metadata class search behaviour

Text-type metadata is included in the search index for two main reasons - and this affects how the metadata is considered when a user makes a search.

  • Display only: The contents of this field will be indexed but is not considered as document content by Funnelback and will not influence the ranking when making a query. The value can be used for display purposes. It can be searched over only when the field is explicitly defined in the search query (e.g. author:shakespeare).

  • Searchable as content: The contents of this field will be considered a part of the document content by Funnelback. Queries will match within this field and the value will contribute to the document’s ranking. The contents of this field will also be automatically considered for spelling and simple auto-completion suggestions. As for display-only metadata the field can be explicitly searched and the content of this field can also be used for display purposes.

11.4. Configuring metadata mappings

Metadata mappings in Funnelback are defined using the administration interface configure metadata mappings tool.

The metadata mapping configuration is accessed from the administer tab in the administration interface.

configuring metadata mappings 01

Running an initial update of your collection is recommended before configuring the metadata mappings. This is because the indexer auto-detects available metadata sources when it scans the content. This in turn makes setting up the metadata mappings a lot simpler because the detected classes can be picked from list of available sources.

The metadata mappings screen lists the metadata mappings that are defined for the current collection. Funnelback’s default configuration also includes commonly-used metadata from HTML and binary file content. Metadata fields from the Dublin Core, AGLS and Open Graph standards have pre-defined mappings as part of the default configuration.

configuring metadata mappings 02

The metadata mappings listing also provides information on the number of documents where the metadata is detected and allows the list to be filtered by keyword, or metadata content type (HTML/XML).

Additional mappings are created by clicking the add new button, and existing mappings can be modified by clicking on the metadata class name in the listing. The metadata mapping editor screen allows configuration of the mapping, allowing multiple metadata sources to be mapped to a single metadata class.

configuring metadata mappings 03

The metadata sources screen allows presents a list of metadata sources that were detected during indexing as well as commonly used HTML tags. In addition, metadata sources can be manually added as a source.

configuring metadata mappings 04
Exercise 27: Configuring metadata
  1. Run a search for death against the silent-films collection that we used in the previous example. A set of results similar to those below will be returned. The results are returned in using Funnelback’s default search results template.

    exercise configuring metadata 01
  2. Open the The car of death search result. The page will load in your browser.

    exercise configuring metadata 02
  3. View the page source and inspect the meta tags in the head region of the document.

    exercise configuring metadata 03
  4. Note down any metadata fields that are listed that contain information that might be useful for the search. Useful information is information that you might use in the search results (e.g. a good description, or URL to a thumbnail image) or keyword metadata (that can be used to contribute to the discoverability of the document). Inspect a few other result pages as well to see if the same fields exist in other pages and if they also seem useful. In the above example the open graph (og) tags appear to have useful content that could add value to the search results listing.

  5. Log in to the administration interface and switch to the silent-films collection.

  6. Open the metadata mappings configuration by selecting configure metadata mappings from the administer tab.

    exercise configuring metadata 04
  7. The currently defined mappings are listed. The default list contains pre-defined mappings of commonly used HTML metadata fields. For many searches this will be sufficient without adding any additional mappings.

    exercise configuring metadata 05
  8. The search result summaries could be enhanced by utilising text sourced from the title, description and image metadata - from tags that we identified (og:title, og:description and og:image). These tags all appear in the current set of mappings - this means that the metadata has already been included in the search index and is available for use in the search results.

  9. In order to use metadata in your search results template four conditions need to be satisfied:

    • The metadata source needs to exist in the content that is being indexed. (e.g. the HTML page needs to have meta tags populated with values).

    • The metadata source needs to be mapped to a Funnelback metadata class.

    • The metadata class needs to be included in the list of summary fields to display (if this is not set anywhere it will default to returning the a, c, p and s metadata classes) and the collection must not have summaries disabled (SM=off).

    • The template needs to be configured to print out the metadata appropriately.

  10. Confirm that the og:title. og:description and og:image metadata sources are mapped to classes by typing each into the filter metadata mappings box above the metadata listing. Make a note of the class names to which these sources are mapped.

  11. Return to the search results for the search for death and view the JSON response (reminder: edit the URL so the search.html reads search.json). Inspect the results element that sits underneath the response element and observe the available result item fields and how they are used in the HTML search results page.

    exercise configuring metadata 06
  12. Inspect the result for the car of death (we previously inspected the HTML source for suitable metadata tags. Observe all the fields that are available within the JSON (or XML) response. Each of these fields can be returned within the result set. Available metadata is returned within the metaData sub element, which is currently displaying metadata from the c metadata class. Only metadata in the c, a, s or p metadata classes will be displayed by default. Recall that there the list of metadata classes returned by Funnelback can be controlled using the summary fields (SF) display option.

  13. Return to the administration interface and select edit collection configuration from the administer tab, then interface from the sidebar. The summary fields value (SF) which takes a comma separated list of fields as the parameter value defines the fields to return (the default value is -SF=[a,c,p,s]). The SF parameter also supports regex expressions that match the metadata field classes. Checking our earlier notes we find that c (description), t (title) and image (thumbnail URL) are the internal classes used for the fields that we are interested in. Add -SF=[c,t,image] to the query processor options. Note: the order of classes in the list is not important, but the comma separated list must sit within square brackets. Save the changes.

    exercise configuring metadata 07
  14. Return to the JSON output and refresh the page. Observe that c, t and image metadata values are now returned. You may notice that the t value appears to be duplicated - this is because there are multiple metadata fields mapped to the same class (e.g. title and og:title), and that value is duplicated across those fields. The image class also contains multiple values (delimited with a vertical bar) as there are multiple fields mapped to the same class (og:image and twitter:image). If you wish to print the value you can strip off the content after the | when printing, or you can map the og:image to a different metadata class.

    exercise configuring metadata 08

11.4.1. Fields containing multiple values

Metadata fields can hold multiple values - this occurs automatically if multiple sources are mapped to the same metadata class and exist in the document being indexed (e.g. description and dc.description may map to the same metadata class, and both fields may appear in a HTML document). When multiple values are stored inside a metadata field they are automatically delimited with a vertical bar character.

A single metadata field within a HTML document can also hold multiple values and the delimiter often varies and may use commas or semicolons. If you have control over the metadata field you should update the CMS templates that generate the metadata fields to use a vertical bar for optimal behaviour in Funnelback.

While it’s possible to change the character(s) used for metadata field delimiters from within Funnelback it is best to change your delimiters to vertical bars as any changes to the characters used to split fields within Funnelback will apply to all metadata fields (so if you choose a comma description fields may start to get split when they shouldn’t). The split characters used for the collection can be changed by editing the -facet_item_sepchars indexer option.

If you require per-field delimiters you can download and configure the metadata delimiters filter from the Funnelback GitHub site.

11.5. Spelling suggestions and simple auto-completion

Metadata field content is used for spelling and simple auto-completion suggestions in the following circumstances:

  • Text type metadata that is searchable as content is added to the general page content and will be considered for spelling and simple auto-complete suggestions if threshold conditions are met.

  • Text metadata that is display only can also be added as an explicit metadata source for spelling and auto-complete suggestions by setting the spelling.suggestion_sources collection configuration option.

11.6. Metadata sources for contextual navigation

Metadata field content can also be used as a source for contextual navigation (or related search) suggestions. By default the keyword and c metadata classes are considered for contextual navigation suggestions. This list of fields can be adjusted by changing the contextual_navigation.summary_fields list within the contextual navigation settings.

The settings are editable from within the administration interface by selecting the administer, then edit collection configuration and choosing contextual navigation from the sidebar (or see the previous exercise on configuring contextual navigation).

Exercise 28: Customise metadata field mappings

In the previous example we discovered that the default mappings for both the title and image result in duplication in the metadata as a result of the web site containing variants of the tags with the same value.

In this exercise we will alter the mappings so that we select one of the fields as the one that will be used for printing and map it to a unique field. We will use the og:title, og:description, og:image and also the og:video values as the fields for display.

  1. Load the administration interface and switch to the silent-films collection.

  2. Edit the metadata mappings (by selecting configure metadata mappings from the administer tab).

  3. Locate the entry for og:description by using the keyword filter. Metadata sources can only be mapped to a single metadata class so it needs to be removed from the existing mapping before being set up as a new class.

    exercise customise metadata field mappings 01
  4. Edit the c metadata class entry by clicking on the row. Five different sources are mapped to the c metadata class - remove the mapping for og:description by locating it in the sources list and clicking on the delete icon that appears when you hover over the row. After removing the source click the save mapping button to update the configuration.

    exercise customise metadata field mappings 02
  5. You are returned to the listing screen. Create a new class called displayText by clicking the add new button. The field should be created as a text type field (it is a description) and be searchable as content (this field contains information that’s useful to consider as content). Click the add new button from the sources section. This opens a window that shows all the detected metadata sources that have not been mapped to anything, along with controls to filter the list. Locate the og:description source and tick the checkbox under the add source column. Multiple sources can be added by checking multiple boxes. Click the save button to add the source.

    exercise customise metadata field mappings 03
  6. Optionally add a description (describing the metadata mapping) before clicking add new mapping.

    exercise customise metadata field mappings 04
  7. Locate the entry for og:title. Update it so that it’s mapped to a class called displayTitle.

  8. Locate the entry for og:image. Update it so that it’s mapped to a class called displayImage.

  9. There isn’t an existing entry for og:video. Add a mapping for og:video to a class named videoUrl. Set this as display only metadata, noting that the value of the URL doesn’t provide any useful information for the purposes of ranking the document. Click the save button once you are done.

  10. The index will need to be rebuilt because the metadata mappings have changed. This is done by running an advanced update to rebuild the index. Return to the main page of the administration interface and select start advanced update from the update tab. Rebuild the index by selecting the reindex live view radio button and clicking on the update button.

    exercise customise metadata field mappings 05
    exercise customise metadata field mappings 06
  11. Once the re-index has completed update the display options so that displayTitle, displayText, displayImage and videoUrl are returned in the search results. Note: don’t remove existing metadata fields. (Hint: amend the SF parameter for the query processor options).

    exercise customise metadata field mappings 07
  12. Search for death again and inspect the JSON or XML output and confirm that the metadata values are returned in the result metadata.

    exercise customise metadata field mappings 08
Exercise 29: Add metadata to search result summaries

In this exercise the template will be configured to print the metadata in the search result summary if available.

  1. Load the administration interface and switch to the silent-films collection.

  2. Edit the template (by selecting design results page from the customise menu)

  3. The default template will load in the editor. This template includes a lot more Freemarker than in previous examples which was stripped back to the bare bones required to format the search results. It is designed to handle most of Funnelback features if the relevant data is returned in the data model. Locate the results block and add the following code to print the metadata. Insert the code immediately after the <cite> tag containing the document URL then save the template (approx line 549).

    <#if s.result.metaData["displayTitle"]??><p>Display title: ${s.result.metaData["displayTitle"]}</p></#if>
    <#if s.result.metaData["displayText"]??><p>Display text: ${s.result.metaData["displayText"]}</p></#if>
    <#if s.result.metaData["displayImage"]??><p><img src="${s.result.metaData["displayImage"]}"/></p></#if>
    exercise add metadata to search results 01
  4. Rerun the search for death and confirm that the metadata and thumbnails are appearing.

    exercise add metadata to search results 02

11.6.1. Review questions

  • Can you map a metadata field to multiple metadata classes?

  • Why would you want to map different metadata fields to the same Funnelback metadata class?

11.6.2. Extended exercises: metadata

  • Change the result formatting so that the thumbnail is displayed to the left of the text.

  • Map the video duration and add this information to the search results summary.

  • Hyperlink the thumbnail image so that it loads the video when clicked upon.

  • Alter the video link to load the video using a JavaScript lightbox.

  • Update the summary fields (SF) expression to use a regular expression to match all the metadata classes that start with display. Hint: the expression to use is display.+.

11.7. Configuring external metadata

External metadata provides administrators with a method of attaching metadata from an external source to documents within a search index based on URL rules.

The metadata defined in the external metadata file is attached to all URLs that start with a specific URL base. This can be used to quickly apply some metadata structure to a site with a human friendly URL scheme.

Any metadata applied this way is accessed in the same manner as standard metadata configured via the metadata mappings configuration tool once the index is built.

Exercise 30: External metadata
  1. Load the administration interface and switch to the silent-films collection.

  2. Navigate to the file manager (select browse collection configuration files from the administer tab).

  3. Create an external_metadata.cfg file by selecting external_metadata.cfg from the select menu that appears at the bottom of the silent-films / config file listing then click on the create button.

    exercise external metadata 01
  4. A blank file editor screen will load.

    External metadata will be defined for a set of specific URLs.

    The first line attaches common metadata to all the documents in the collection - type=Silent film. This is applied to all documents that have a URL starting with https://archive.org/details (which is every document in this collection).

    The second line attaches common metadata of director=Charlie Chaplin to all URLs that start with https://archive.org/details/CC_ which is a subset of films featuring Charlie Chaplin.

    The rest of the lines attach specific metadata to the URLs listed - again all these URLs are treated as left matching - but as they are specific the pattern will only match a single URL.

    Copy and paste the following into the editor and click save:

    https://archive.org/details filmType:"Silent film"
    https://archive.org/details/CC_ director:"Charlie Chaplin"
    https://archive.org/details/Downhill_1927 director:"Alfred Hitchcock" year:1927 language:English runningTime:82
    https://archive.org/details/EasyVirtue1928 director:"Alfred Hitchcock" year:1928 language:English runningTime:85
    https://archive.org/details/CannedHarmony director:"Alice Guy Blaché" year:1912 runningTime:13
    https://archive.org/details/TheBurstrupHomesMurderCase director:"Alice Guy Blaché" year:1911 runningTime:18
  5. When defining new fields as external metadata it is good practice to ensure any external metadata fields have metadata classes defined in the metadata class definitions. This allows you to see what fields are defined for the collection and more importantly to tell the indexer what type of metadata the field values contain. Edit the metadata mappings (select edit metadata mappings from the administer tab) and add the following definitions to the mappings:

    Metadata class name Source Type

    filmType

    EXTERNAL_METADATA_filmType

    Text (display only)

    director

    EXTERNAL_METADATA_director

    Text (Searchable as content)

    year

    EXTERNAL_METADATA_year

    Text (display only)

    runningTime

    EXTERNAL_METADATA_runningTime

    Text (display only)

    language

    EXTERNAL_METADATA_language

    Text (display only)

    When defining the mappings ensure the class name matches the fieldname used in the external_metadata.cfg exactly.

    When adding the field mapping add the class as for a normal metadata field, but when adding the source(s) to the mapping you will need to type in the source name instead of picking from a list of detected sources. The actual value you type here doesn’t really matter - best practice is to using a label similar to EXTERNAL_METADATA_<field_name> where <field_name> corresponds to the field from the external metadata file. e.g. setting up a mapping for the filmType field (appearing as filmType:value in the external metadata file) would look similar to:

    exercise external metadata 02

    Observe the typed value of the name (filmType):

    exercise external metadata 03

    Add metadata entries for the other fields sourced from external metadata. You will find that language is already defined - for this field add the source to the existing field instead of creating a new field.

  6. Rebuild the index. (Select start advanced update from the update tab, then select re-index live view and click update).

  7. Add the display options to add the new metadata classes to the summary fields (-SF=[displayText,displayTitle,displayImage,videoUrl,filmType,director,runningTime,language,year] (Edit collection settings from the administer tab, then switch to the interface screen and add these values to the query processor options).

  8. Run a search for hitchcock against the silent-films collection and view the JSON page source and confirm that additional metadata is returned in the data model with the result summaries (edit the URL to change search.html to search.json).

    exercise external metadata 04

12. Search result filters (faceted navigation)

Faceted navigation can be used to filter or refine a set of search results, allowing users to quickly drill down on items or sub-sections of interest within a search that may contain hundreds of thousands of results.

Faceted navigation is designed to help users refine their search by providing a set of filters that are applicable to the set of search results returned by the search query.

Facets contain categories and the default display shows a count of matching results for each category. Categories without any matching results are hidden, preventing users from running a search that returns no results.

12.1. Facets and category values

A facet represents a way in which your documents can be categorised (e.g. job, location or author).

Category values are the individual values that can be filtered on.

e.g. For a facet holding various authors: the author is the facet while Ada, Alan and Bob are the category values that the user might select to restrict the documents to.

12.2. Types of facets

Funnelback supports five types of facets. Each type allows a user to filter the result set.

The types differ by proving choice in how the filtering is applied - allowing filtering on single or multiple categories and providing options on how the filters are combined and how they apply to the result set.

types of facets 01

12.2.1. Filter on a single category

Allows a user to filter the result set using a single category value chosen from the facet. Upon selection the facet displays only the selected category value.

12.2.2. Filter on a single category, with drill down

This is sometimes referred to as a hierarchical facet.

Allows a user to filter the result set using a single category value chosen from the facet. After the filter is applied sub-categories may be displayed for the facet.

12.2.3. Filter on multiple categories

This is sometimes referred to as multi-select or checkbox faceting.

Allows a user to filter the result set using multiple categories from the same facet. The categories can be combined using either AND or OR logic.

12.2.4. Toggle between categories

Sometimes also referred to as a radio facet.

Allows a user to filter the result set on a single category. The set of categories remain after the filter is applied allowing a user to toggle through the different categories.

12.2.5. Toggle between tabs

This is a special type of facet that is presented as a series of tabs. It is designed to allow a search to be separated into a series of tabs, with other facets then being used to further filter the result set. Because of this the tab facet isn’t

Exercise 31: Add a facet

In this exercise a filter on single category facet, with categories sourced from an existing metadata field will be configured for the silent-films collection. Use of metadata as a category source requires metadata to be configured for the collection prior to faceted navigation being configured.

Before you configure faceted navigation you need to work out what you will use for the faceted navigation - in this case we’ll use the metadata fields that we configured via external metadata (director, language, year, etc.).

  1. Load the administration interface and switch to the silent-films collection.

  2. Open the faceted navigation editor by selecting customise faceted navigation from the customise tab

    exercise add a facet 02
  3. Create a new facet by clicking on the add new button. A facet is a filter that will be returned with the search results.

    exercise add a facet 03
  4. The new facet screen allows you to choose from 6 different types of facets. Hovering over the images provides some immediate feedback on how the facet behaves. Click on the filter on a single category tile.

    exercise add a facet 04
  5. Enter Director as the name for the facet.

    exercise add a facet 05
  6. The category source then needs to be selected by choosing an item from the drop down that appears beneath the category values heading. Select predefined values contained in a metadata or XML field (metadata) from the list and enter director (the metadata class name that you defined in the collection metadata mappings) as the value for the metadata field. Note that metadata field names are case sensitive and must exactly match the class name that is defined in the mappings. Click add.

    exercise add a facet 06
  7. The facet listing screen updates to display the director facet. Observe that the facet is currently unpublished and there is a button available to publish the facet definition.

    exercise add a facet 07
  8. Add a second facet called Release year that sources the metadata values from the year metadata class. This time click the add and publish button.

    exercise add a facet 08
  9. Run a search for !showall against the live version of the silent-films collection and observe that there is one facet (Release year) and the values returned for each facet are the individual values that are defined for the metadata fields. Observe that the Director facet is not displaying. This is because the facet has not been published. If you repeat the query on the preview version of the search you will see both facets.

  10. Return to the faceted navigation editor and click the publish button for the director facet. Repeat the live search above and observe that both facets now display.

  11. Click unpublish on the release year facet and rerun the live search observing that the director facet is now the only facet returned.

  12. Faceted navigation counts the different values it finds - so for a facet to be effective you need to make sure your metadata field is populated with consistent values - where the same values are used on many items (such as a controlled list of keywords).

    exercise add a facet 09
  13. Observe that the facet categories presented are all lower case - this is because Funnelback treats the keywords as case insensitive when counting the facets (i.e. Hitchcock = hitchcock). The capitalisation can be preserved to match the metadata field values, but be aware that you may start to see duplicate terms if the capitalisation of the keywords doesn’t match for all pages. A display option must be set to make the facet categories case sensitive. In the administration interface select edit collection configuration from the administer tab, then select interface from the sidebar. Add -rmc_sensitive=true to the list of query processor options and then click save.

  14. Reload the search results page and observe that all the categories are now case sensitive.

    exercise add a facet 10

12.3. Facet categories

Facet category values are the individual filter choices that appear beneath a facet.

Each facet contains one or more facet categories. Selecting a facet category will filter the search results on include only results that include the category (subject to any other selected categories and the matching logic).

12.3.1. Category sources

Facet categories are sourced from a number of different things. Understanding how the facets and facet categories relate to the source data is important for creating effective faceted navigation (and for planning site redevelopments to make use of facets).

Categories based on metadata

A facet that has category values sourced from predefined values contained within a metadata or XML field (metadata) has a 1:1 relationship between the facet and a Funnelback metadata class.

The categories that can appear beneath a metadata facet are all the unique values found within the metadata class. The count for each category is the number of unique times that the metadata value is found within the result set.

Categories based on URL patterns or gscopes

A facet that has category values sourced from predefined URL sets (generalised scopes or gscopes) or a set of URL patterns (URL pattern) has a 1:1 relationship between each category and a defined URL pattern set or generalised scope.

The category label is determined by the label assigned when creating the facet.

The count for each category is the number of items in the result set that belong to the defined URL pattern set or gscope.

Generalised scopes are used to define sets of URLs and are covered in FUNL202 - Funnelback for advanced implementers.
Categories based on URL path or directory structure

A facet that has category values sourced from the item’s location in a URL path or directory structure (URL fill) has a 1:1 relationship between the facet and a defined root URL or folder.

The categories are determined by splitting the URL on the folder structure (with folder names as the possible categories) to a determined depth. The depth defines a hierarchical structure to apply to the facet.

The count for each category is the number of results that sit beneath the folder.

Categories based on date

A facet that has category values sourced from groupings based on the item’s date (dynamic date fill) has a 1:1 relationship between the facet and the document’s indexed date value.

The categories that are determined from hardcoded groupings of dates and the counts are based on the number results with a date value such as yesterday, last week, last month, 2014.

Categories based on results from a query

A facet that has category values sourced from documents which match a particular query has a 1:1 relationship between each category and the set of URLs retuned for the defined query.

The category label is determined by the label assigned when creating the facet.

The count for each category is the number of items in the result set that are in the set of results returned for the defined query.

Categories based on source collection

A facet that has category values sourced from documents which belong to a particular collection has a 1:1 relationship between each category and membership to sets of collections.

The category label is determined by the label assigned when creating the facet.

The count for each category is the number of items in the result set that belong to each defined set of collections.

Categories based on numerical range

A facet that has category values sourced from documents whose metadata value lies within a numerical range has a 1:1 relationship between each category and the number of results returned that belong to each defined range.

The category label is determined by the label assigned when creating the facet.

The count for each category is the number of items in the result set that are in the each of the pre-defined ranges.

12.3.2. Category counts

The counts displayed for categories are estimates that are calculated based on the result set. Because of this the accuracy will reduce as the search index becomes larger. The numbers may also change when a facet is selected as the estimates are recalculated every time the result set is produced. The result set is produced based on a daat limit of 5000 - which contains the first 5000 matching results found in the index.

12.3.3. Sorting category values

Facet category values can be sorted by a number of different attributes:

  • count: the category values are sorted by the category count, either largest to smallest, or smallest to largest.

  • label: the category values are sorted by the label, either alphabetically (A-Z) or reverse alphabetically (Z-A).

  • numeric label: the category values are sorted by the label, but with numbers interpreted numerically. e.g. a numeric label sort would sort as follows: 1, 10, 11, 100, compared with a label sort which would sort: 1, 10, 100, 11.

  • date: the category values are sorted by date, either most recent to oldest, or oldest to most recent.

  • selected values first: selected category values are placed above unselected category values.

  • configuration order: for facet types where the individual category values are defined, the order in which they appear in the configuration will be the order that is used to display the facets. The order can be changed by dragging the items around in the configuration.

  • custom sort logic: custom sort logic, implemented as a Groovy comparator, is used to sort the category values. Can be used to apply custom sort ordering to metadata based facets.

12.3.4. Renaming category values

Categories can be renamed by using a Groovy hook script.

12.3.5. Blacklisting and whitelisting category values

Two collection configuration options can be used to restrict the category values that can be displayed within a facet.

  • faceted_navigation.black_list: category values listed will be removed from the facet if they are returned in the list of facets.

  • faceted_navigation.white_list: only the category values listed will be displayed if they are returned in the list of facets. Other category values are removed.

Exercise 32: Query based faceted navigation

In this exercise query based facets will be configured for the silent-films collection from previous exercises.

  1. Load the administration interface and switch to the silent-films collection.

  2. Open the faceted navigation editor by selecting customise faceted navigation from the customise tab

  3. Create a new filter on single category facet by clicking on the add facet button then selecting the filter on single category tile. The two metadata facets from the previous example will be displayed in the editor.

  4. Enter Director (query) as the name for the facet and select documents which match a particular query from the category values are sourced from dropdown. Enter "charlie chaplin" (including the double quotes) into the query field and Charlie Chaplin into the label field. The double quotes ensure that the query is run as a phrase. The value inside the label field is what the user will see as a choice to click on.

    exercise query based faceted navigation 01
    A re-index is required after adding or changing any query based faceted navigation categories.
  5. Add a second category by clicking the + button next to the Director (query) facet. Enter a second item for Buster Keaton using "buster keaton" as the query and Buster Keaton as the label.

    exercise query based faceted navigation 02
  6. Then select add and publish from the dropdown on the add button.

    exercise query based faceted navigation 03
  7. Return to the administration home page and rebuild the index by running the re-apply gscopes to the live view option from the advanced update screen. This is required to generate the generalised scopes that underpin the categories for the query-based facets. Running a re-index of the live view will also regenerate the facets but may take longer as a number of other operations are run as part of the re-index.

    Gscopes (or generalised scopes) allow an administrator to define sets of pages based on the URL matching a pattern or query that can be used for query scoping, profiles and faceted navigation. Gscopes are covered in detail in the FUNL202 course.
  8. Run a search for !showall against the silent-films collection and observe that there is an additional facet Director (query) and the values returned for each facet are the two categories that you just defined. Faceted navigation for query categories (and predefined URL set/gscope based categories) requires a separate entry for each category that will counted. The counts refer to the number of items in the result set that match the query defined in the facet configuration. This differs from metadata facets that count all the unique metadata values that appear within a metadata field. For this reason query and gscope facets are more difficult for an administrator to maintain as the faceted navigation configuration must be updated every time a new category is added. For metadata based facets new categories that are added to an existing metadata field will automatically appear.

    exercise query based faceted navigation 04
Extended exercises: Faceted navigation
  1. Create a dates field fill facet that builds a facet based on the document dates.

  2. Recreate the director facet as a filter on multiple categories facet.

  3. Update the template code to apply an ordering to the facets. (hint: instructions are provided in the documentation link below)

  4. Change the sort order for the release year to be alphabetic

13. Using remote includes

Website owners spend a lot of time crafting the look and feel of their templates and pages. The styles and templates used for website pages can be applied to the Funnelback search results pages without the need for a front-end developer to remember to manually update the search template whenever a change to a site’s look and feel occurs.

It is very important to plan the remote includes carefully so that the search results page returned by Funnelback is of a valid page structure when all the remote code is included.

The best way to plan this is to ensure each include is a distinct and well-formed code chunk (such as you’d nest inside a <div>).

Funnelback’s remote inclusion requires information on where to capture the include from within the remote page. This can be specified either with a start and end regular expression, or by CSS selectors. CSS selectors are preferred if the source page is well structured. Where start and end patterns are requried the best way to make this work effectively and be easy to maintain is to add additional HTML comments to your page that you can use to mark where each include starts and ends.

The remote include is implemented in a macro: IncludeUrl. This macro takes a number of parameters including:

  • url: the URL of the remote page to include

  • start: string of text where the included text should start. Any double quote characters should be escaped with a backslash as the string is a Java regular expression.

  • end: string of text where the included text should end. Any double quote characters should be escaped with a backslash as the string is a Java regular expression.

  • expiry: number of seconds to cache the response

  • convertrelative: whether or not to convert relative links to absolute ones.

  • cssSelector:

  • removeByCssSelectors:

Full documentation for IncludeUrl is available in the Funnelback documentation (and includes additional parameters)

Exercise 33: Remotely include header and styling for a search results template

This exercise will apply the look and feel of the Internet Archive’s Silent Films collection to the search that we have created for this content.

  1. Browse to the silent films archive at https://archive.org/details/silent_films and view the source code. Identify appropriate start and end points within the markup of this page for the page header. Use of web developer tools built into your web browser will assist in identifying appropriate start and end points. An appropriate header include for the silent films site starts with the

    <a href="#maincontent" class="hidden-for-screen-readers">Skip to main content</a>

    immediately after the <head> tag and includes all the HTML code up to the <!--/.container--> comment in the code.

  2. Edit the template for the silent-films collection in the Funnelback administration interface. (Design results page from the customise tab). Add the following code immediately after the <body> tag (approx line 55). Note the remote include specifies an expiry of 1 second to assist with testing, and double quotes contained within the start and end parameters have a backslash placed in front of them.

    <!--start included header-->
    <@fb.IncludeUrl url="https://archive.org/details/silent_films" start="<a href=\"#maincontent\" class=\"hidden-for-screen-readers\">Skip to main content</a>" end="<!--/.container-->" expiry=1 convertrelative=true/>
    <!--end included header-->
    exercise remotely include header and styling for a search results template 01
  3. Save and publish the template then run a search for death against the silent-films collection. Observe that the header code is now included in the template, but the styling is not correct.

    exercise remotely include header and styling for a search results template 02
  4. Inspect the source code for the silent films archive again and look for an appropriate include for JavaScript and styles from the header block. An appropriate include is the region of markup in the source HTML that you wish to include. If working with your own site template you should locate all the things to include together and add unique HTML comments at the start and end of each region you wish to include. This makes the IncludeUrl call much simpler to define, and also provides a reminder in the template that this region is used by Funnelback. An appropriate include for the silent films archive is everything in the header starting from

    <script src="//archive.org/includes/jquery-1.10.2.min.js

    until

    <link rel="SHORTCUT ICON" href="https://archive.org/images/glogo.jpg”/>`
    exercise remotely include header and styling for a search results template 03
  5. Edit the template again and this time add another include to the document head region to include the highlighted code from the screenshot above. Add the following code immediately after the <title> tag.

    <!--start document header include-->
    <@fb.IncludeUrl url="https://archive.org/details/silent_films" start="<script src=\"//archive.org/includes/jquery-1.10.2.min.js" end="<link rel=\"SHORTCUT ICON\" href=\"https://archive.org/images/glogo.jpg\"/>" expiry=1 convertrelative=true />
    <!--end document header include-->
    exercise remotely include header and styling for a search results template 04
  6. The Funnelback default template has a reliance on JQuery and Bootstrap, loaded at the bottom of the template. We’ll need to remove these or they will conflict with the versions that have just been imported. Resolving conflicts with JQuery and styles is the most challenging part of integrating the search results template. Applying your own styling to the search results will reduce the conflicts. Locate the block of JavaScript imports at the bottom of the template and comment out the first three that have been imported as part of the IncludeUrl call. Update the block of code as below. Note that Freemarker comments

    <#-- This is a Freemarker comment -->

    have been used instead of HTML comments

    <!-- This is a HTML comment -->

    This means the commented code will be removed from what is returned to the user. Once you have commented out the lines, save and publish.

    exercise remotely include header and styling for a search results template 05
  7. Refresh the search results and observe that the results page looks more correct now (icon styles are now showing) but still isn’t formatting the header section properly.

    exercise remotely include header and styling for a search results template 06
  8. Look for any missing style sheets or element styles. The web developer tools will also assist in identifying the problem. Observe that the body tag has some additional styles applied. Try adding these styles to the Funnelback template’s body tag

    <body class="navia ia-module lists showdetails">
    exercise remotely include header and styling for a search results template 07
  9. Refresh the search results page - this time the results look almost fully correct. The minor style issues can be resolved by in page or included styles to override the problems, or by adapting the included stylesheet (if you are in control of the included stylesheet).

    exercise remotely include header and styling for a search results template 08
  10. Once the page is working correctly adjust the expiry values applied to each include. We set them to expire after 1 second in the examples above - this should be changed to a longer value such as a day (86400 seconds) so that the includes are not fetched each time the page is requested. Using cached versions of the includes will speed up the page load time.

Review exercises
  1. Set up and configure services for the silent films collection.

  2. Complete styling of the template for the inventors collection