Appendix A Schema specification and tag documentation

<content>
 <sequence minOccurs="1" maxOccurs="1">
  <elementRef key="params"/>
  <elementRef key="rules" minOccurs="0"/>
  <elementRef key="contexts" minOccurs="0"/>
  <elementRef key="excludes" minOccurs="0"/>
  <elementRef key="filters" minOccurs="0"/>
 </sequence>
</content>
    ⚓

element config
{
   attribute version { text }?,
   ( params, rules?, contexts?, excludes?, filters? )
}⚓

• att.match
  • @match
• att.labelled
  • @label

When the indexer is extracting keyword-in-context strings for each word, it uses a common-sense approach based on common element definitions, so that for example when it reaches the end of a paragraph, it will not continue into the next paragraph to get more context words. You may have special runs of text in your document collection which do not appear to be bounding contexts, but actually are; for example, you may have span elements with class=note that appear in the middle of sentences but are not actually part of them. Use <context> elements to identify these special contexts so that the indexer knows the right boundaries from which to retrieve its keyword-in-context strings.

<content>
 <empty/>
</content>
    ⚓

element context
{
   att.match.attributes,
   att.labelled.attributes,
   attribute context { text }?,
   empty
}⚓

<content>
 <elementRef key="context" minOccurs="0"
  maxOccurs="unbounded"/>
</content>
    ⚓

element contexts { context* }⚓

Note that contexts are necessary for phrasal searching or wildcard searching.

<content>
 <empty/>
</content>
    ⚓

element createContexts
{
   (
      ( attribute create { "false" }? )
    | (
         attribute create { "true" }?,
         attribute phrasalSearch { text }?,
         attribute wildcardSearch { text }?,
         attribute maxKwicsToHarvest { text }?,
         attribute maxKwicLength { text }?,
         attribute kwicTruncateString { text }?
      )
   ),
   empty
}⚓

The indexing process checks each word as it builds the index, and keeps a record of all words which are not found in the configured dictionary. Though this does not have any direct effect in the indexing process, all words not found in the dictionary are listed in the staticSearch report (see 8.9 Generated report). This can be very useful: all words listed are either foreign (not part of the language of the dictionary) or perhaps misspelled (in which case they may not be correctly stemmed and index, and should be corrected).

staticSearch provides a default dictionary in xsl/english_words.txt that can be copied and adapted if working in English; lots of dictionaries for other languages are available on the Web.

<content>
 <empty/>
</content>
    ⚓

element dictionary { attribute file { text }, empty }⚓

• att.match
  • @match

<exclude> can be used to identify documents or parts of documents that are to be omitted from indexing, but, unlike setting <weight> to zero, should be retained during the indexing process. This is helpful in cases where the text itself should be ignored by the indexer, but should still appear in KWICs. Another common use is for multiple search engines/pages that each have their own special features; in this case, you may want one specific search index/page to ignore filter controls (HTML <meta> elements, as described in 5 Search facet features) which are provided to support other search pages.

<content>
 <empty/>
</content>
    ⚓

element exclude
{
   att.match.attributes,
   attribute type { "index" | "filter" },
   empty
}⚓

<content>
 <elementRef key="exclude" minOccurs="1"
  maxOccurs="unbounded"/>
</content>
    ⚓

element excludes { exclude+ }⚓

<content>
 <elementRef key="span" minOccurs="1"
  maxOccurs="unbounded"/>
</content>
    ⚓

element filter { attribute filterName { text }, span+ }⚓

<content>
 <elementRef key="filter" minOccurs="1"
  maxOccurs="unbounded"/>
</content>
    ⚓

element filters { filter+ }⚓

When the staticSearch build process creates its output, many files need to be added to the website for which an index is being created. For convenience, all of these files are stored in a single folder. This element is used to specify the name of that folder. The default is staticSearch, but if you would prefer something else, you can specify it here. You may also use this element if you are defining two different searches within the same site, so that their files are kept in different locations.

<content>
 <empty/>
</content>
    ⚓

element output { attribute dir { text }, empty }⚓

<content>
 <elementRef key="searchPage"/>
 <elementRef key="index" minOccurs="0"/>
 <elementRef key="stopwords" minOccurs="0"/>
 <elementRef key="dictionary" minOccurs="0"/>
 <elementRef key="tokenizer" minOccurs="0"/>
 <elementRef key="scoringAlgorithm"
  minOccurs="0"/>
 <elementRef key="stemmer" minOccurs="0"/>
 <elementRef key="createContexts"
  minOccurs="0"/>
 <elementRef key="results" minOccurs="0"/>
 <elementRef key="version" minOccurs="0"/>
 <elementRef key="output"/>
</content>
    ⚓

element params {  }⚓

• att.match
  • @match

The rule element is used to identify nodes in the XHTML document collection which should be treated in a special manner when indexed; either they might be ignored (if weight=0), or any words found in them might be given greater weight than words in normal contexts weight>1. Words appearing in headings or titles, for example, might be weighted more heavily, while navigation menus, banners, or footers might be ignored completely.

<content>
 <empty/>
</content>
    ⚓

element rule { att.match.attributes, attribute weight { text }, empty }⚓

<content>
 <elementRef key="rule" minOccurs="1"
  maxOccurs="unbounded"/>
</content>
    ⚓

element rules { rule+ }⚓

<scoringAlgorithm> is an optional element that specifies which scoring algorithm to use when calculating the score of a term and thus the order in which the results from a search are sorted.

<content>
 <empty/>
</content>
    ⚓

element scoringAlgorithm
{
   attribute name { "raw" | "tf-idf" | "BM25" | "BM25L" }?,
   empty
}⚓

The search page is a regular HTML page which forms part of your site. The only important characteristic it must have is a <div> element with id=staticSearch, whose contents will be rewritten by the staticSearch build process. See 8.6 Creating a search page.

<content>
 <empty/>
</content>
    ⚓

element searchPage { attribute file { text }, empty }⚓

<content>
 <alternate minOccurs="1"
  maxOccurs="unbounded">
  <anyElement require="http://www.w3.org/1999/xhtml"
   minOccurs="0" maxOccurs="unbounded"/>
  <textNode/>
 </alternate>
</content>
    ⚓

element span { attribute lang { text }?, ( anyElement_span_1* | text )+ }⚓

The staticSearch project currently has only two real stemmers: an implementation of the Porter 2 algorithm for modern English, and an implementation of the French Snowball stemmer. These appear in the /stemmers/en/ and /stemmers/fr/ folders. The default value for this parameter is en; to use the French stemmer, use fr. We will be adding more stemmers as the project develops. However, if your document collection is not English or French, you have a couple of options, one hard and one easy.

• Hard option: implement your own stemmers. You will need to write two implementations of the stemmer algorithm, one in XSLT (which must be named ssStemmer.xsl) and one in JavaScript (ssStemmer.js), and confirm that they both generate the same results. The XSLT stemmer is used in the generation of the index files at build time, and the JavaScript version is used to stem the user's input in the search page. You can look at the existing implementations in the /stemmers/en/ folder to see how the stemmers need to be constructed. Place your stemmers in a folder called /stemmers/[yourlang]/, and specify yourlang in the configuration file.
• Easy option: Use the identity stemmer (which is equivalent to turning off stemming completely), and make sure wildcard searching is turned on. Then your users can search using wildcards instead of having their search terms automatically stemmed. To do this, specify the value identity in your configuration file.

Another alternative is the stripDiacritics stemmer. Like the identity stemmer, this is not really a stemmer at all; what it does is to strip out all combining diacritics from tokens. This is a useful approach if you document collection contains texts with accents and diacritics, but your users may be unfamiliar with the use of diacritics and will want to search just with plain unaccented characters. For example, if a text contains the word élève, but you would like searchers to be able to find the word simply by typing the ascii string eleve, then this is a good option. Combined with wildcards, it can provide a very flexible and user-friendly search engine in the absence of a sophisticated stemmer, or for cases where there are mixed languages so a single stemmer will not do. To use this option, specify the value stripDiacritics in your configuration file.

<content>
 <empty/>
</content>
    ⚓

element stemmer
{
   attribute dir
   {
      "stemmers/en/"
    | "stemmers/fr/"
    | "stemmers/identity"
    | "stemmers/stripDiacritics"
   },
   empty
}⚓

A stopword is a word that will not be indexed, because it is too common (the, a, you and so on). There are common stopwords files for most languages available on the Web, but it is probably a good idea to take one of these and customize it for your project, since there will be words in the website which are so common that it makes no sense to index them, but they are not ordinary stopwords. For example, in a website dedicated to the work of John Keats, the name keats should probably be added to the stopwords file, since almost every page will include it, and searching for it will be pointless.

staticSearch provides a default set of common stopwords for English, which you'll find in xsl/english_stopwords.txt. One way to find appropriate stopwords for your site is to generate your index, then search for the largest JSON index files that are generated, to see if they might be too common to be useful as search terms. You can also use the Word Frequency table in the generated staticSearch report (see 8.9 Generated report).

<content>
 <empty/>
</content>
    ⚓

element stopwords { attribute file { text }, empty }⚓

<version> enables you to specify the path to a plain-text file containing a simple version number for the project. This might take the form of a software-release-style version number such as 1.5, or it might be a Subversion revision number or a Git commit hash. It should not contain any spaces or punctuation. If you provide a version file, the version string will be used as part of the filenames for all the JSON resources created for the search. This is useful because it allows the browser to cache such resources when users repeatedly visit the search page, but if the project is rebuilt with a new version, those cached files will not be used because the new version will have different filenames. The path specified is relative to the location of the configuration file (or absolute, if you wish).

<content>
 <empty/>
</content>
    ⚓

element version { attribute file { text }, empty }⚓

<content>
 <valList>
  <valItem ident="true"/>
  <valItem ident="false"/>
 </valList>
</content>
    ⚓

ssdata.boolean = "true" | "false"⚓