docs: Add sphinx_selective_exclude extension suite.

Designed specifically to workaround issues we were facing with generating
multiple conditionalized output docsets from a single master doctree.
Extensions were factored out into a separate project, based on the fact
that many other Sphinx users experience similar or related problems:
https://github.com/pfalcon/sphinx_selective_exclude

Corresponds to the 182f4a8da57 upstream revision.

6 files changed, 317 insertions, 0 deletions

diff --git a/docs/sphinx_selective_exclude/LICENSE b/docs/sphinx_selective_exclude/LICENSE
new file mode 100644
index 0000000000..0b47ced8a1
--- /dev/null
+++ b/docs/sphinx_selective_exclude/LICENSE
@@ -0,0 +1,25 @@
+Copyright (c) 2016 by the sphinx_selective_exclude authors.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+* Redistributions of source code must retain the above copyright
+  notice, this list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright
+  notice, this list of conditions and the following disclaimer in the
+  documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/docs/sphinx_selective_exclude/README.md b/docs/sphinx_selective_exclude/README.md
new file mode 100644
index 0000000000..cc9725c21c
--- /dev/null
+++ b/docs/sphinx_selective_exclude/README.md
@@ -0,0 +1,138 @@
+Sphinx eager ".. only::" directive and other selective rendition extensions
+===========================================================================
+
+Project home page: https://github.com/pfalcon/sphinx_selective_exclude
+
+The implementation of ".. only::" directive in Sphinx documentation
+generation tool is known to violate principles of least user surprise
+and user expectations in general. Instead of excluding content early
+in the pipeline (pre-processor style), Sphinx defers exclusion until
+output phase, and what's the worst, various stages processing ignore
+"only" blocks and their exclusion status, so they may leak unexpected
+information into ToC, indexes, etc.
+
+There's multiple issues submitted upstream on this matter:
+
+* https://github.com/sphinx-doc/sphinx/issues/2150
+* https://github.com/sphinx-doc/sphinx/issues/1717
+* https://github.com/sphinx-doc/sphinx/issues/1488
+* etc.
+
+They are largely ignored by Sphinx maintainers.
+
+This projects tries to rectify situation on users' side. It actually
+changes the way Sphinx processes "only" directive, but does this
+without forking the project, and instead is made as a standard
+Sphinx extension, which a user may add to their documentation config.
+Unlike normal extensions, extensions provided in this package
+monkey-patch Sphinx core to work in a way expected by users.
+
+eager_only
+----------
+
+The core extension provided by the package is called `eager_only` and
+is based on the idea by Andrea Cassioli (see bugreports above) to
+process "only" directive as soon as possible during parsing phase.
+This approach has some drawbacks, like producing warnings like
+"WARNING: document isn't included in any toctree" if "only" is used
+to shape up a toctree, or the fact that changing a documentation
+builder (html/latex/etc.) will almost certainly require complete
+rebuild of documentation. But these are relatively minor issues
+comparing to completely broken way "only" works in upstream Sphinx.
+
+modindex_exclude
+----------------
+
+"only" directive allows for fine-grained conditional exclusion, but
+sometimes you may want to exclude entire module(s) at once. Even if
+you wrap an entire module description in "only" directive, like:
+
+    .. only: option1
+        .. module:: my_module
+    
+        ...
+
+You will still have an HTML page generated, albeit empty. It may also
+go into indexes, so will be discoverable by users, leading to less
+than ideal experience. `modindex_exclude` extension is design to
+resolve this issue, by making sure that any reference of a module
+is excluded from Python module index ("modindex"), as well as
+general cross-reference index ("genindex"). In the latter case,
+any symbol belong to a module will be excluded. Unlike `eager_only`
+extension which appear to have issued with "latexpdf" builder,
+`modindex_exclude` is useful for PDF, and allows to get cleaner
+index for PDF, just the same as for HTML.
+
+search_auto_exclude
+-------------------
+
+Even if you exclude soem documents from toctree:: using only::
+directive, they will be indexed for full-text search, so user may
+find them and get confused. This plugin follows very simple idea
+that if you didn't include some documents in the toctree, then
+you didn't want them to be accessible (e.g. for a particular
+configuration), and so will make sure they aren't indexed either.
+
+This extension depends on `eager_only` and won't work without it.
+Note that Sphinx will issue warnings, as usual, for any documents
+not included in a toctree. This is considered a feature, and gives
+you a chance to check that document exclusions are indeed right
+for a particular configuration you build (and not that you forgot
+to add something to a toctree).
+
+Summary
+-------
+
+Based on the above, sphinx_selective_exclude offers extension to let
+you:
+
+* Make "only::" directive work in an expected, intuitive manner, using
+  `eager_only` extension.
+* However, if you apply only:: to toctree::, excluded documents will
+  still be available via full-text search, so you need to use
+  `search_auto_exclude` for that to work as expected.
+* Similar to search, indexes may also require special treatment, hence
+  there's the `modindex_exclude` extension.
+
+Most likely, you will want to use all 3 extensions together - if you
+really want build subsets of docimentation covering sufficiently different
+configurations from a single doctree. However, if one of them is enough
+to cover your usecase, that's OK to (and why they were separated into
+3 extensions, to follow KISS and "least surprise" principles and to
+not make people deal with things they aren't interested in). In this case,
+however remember there're other extensions, if you later hit a usecase
+when they're needed.
+
+Usage
+-----
+
+To use these extensions, add https://github.com/pfalcon/sphinx_selective_exclude
+as a git submodule to your project, in documentation folder (where
+Sphinx conf.py is located). Alternatively, commit sphinx_selective_exclude
+directory instead of making it a submodule (you will need to pick up
+any project updates manually then).
+
+Add following lines to "extensions" settings in your conf.py (you
+likely already have some standard Sphinx extensions enabled):
+
+    extensions = [
+        ...
+        'sphinx_selective_exclude.eager_only',
+        'sphinx_selective_exclude.search_auto_exclude',
+        'sphinx_selective_exclude.modindex_exclude',
+    ]
+
+As discussed above, you may enable all extensions, or one by one.
+
+Please note that to make sure these extensions work well and avoid producing
+output docs with artifacts, it is IMPERATIVE to remove cached doctree if
+you rebuild documentation with another builder (i.e. with different output
+format). Also, to stay on safe side, it's recommended to remove old doctree
+anyway before generating production-ready documentation for publishing. To
+do that, run something like:
+
+    rm -rf _build/doctrees/
+
+A typical artificat when not following these simple rules is that content
+of some sections may be missing. If you face anything like that, just
+remember what's written above and remove cached doctrees.
diff --git a/docs/sphinx_selective_exclude/__init__.py b/docs/sphinx_selective_exclude/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
--- /dev/null
+++ b/docs/sphinx_selective_exclude/__init__.py
diff --git a/docs/sphinx_selective_exclude/eager_only.py b/docs/sphinx_selective_exclude/eager_only.py
new file mode 100644
index 0000000000..82766c2e64
--- /dev/null
+++ b/docs/sphinx_selective_exclude/eager_only.py
@@ -0,0 +1,45 @@
+#
+# This is a Sphinx documentation tool extension which makes .only::
+# directives be eagerly processed early in the parsing stage. This
+# makes sure that content in .only:: blocks gets actually excluded
+# as a typical user expects, instead of bits of information in
+# these blocks leaking to documentation in various ways (e.g.,
+# indexes containing entries for functions which are actually in
+# .only:: blocks and thus excluded from documentation, etc.)
+# Note that with this extension, you may need to completely
+# rebuild a doctree when switching builders (i.e. completely
+# remove _build/doctree dir between generation of HTML vs PDF
+# documentation).
+#
+# This extension works by monkey-patching Sphinx core, so potentially
+# may not work with untested Sphinx versions. It tested to work with
+# 1.2.2 and 1.4.2
+#
+# Copyright (c) 2016 Paul Sokolovsky
+# Based on idea by Andrea Cassioli:
+# https://github.com/sphinx-doc/sphinx/issues/2150#issuecomment-171912290
+# Licensed under the terms of BSD license, see LICENSE file.
+#
+import sphinx
+from docutils.parsers.rst import directives
+
+
+class EagerOnly(sphinx.directives.other.Only):
+
+    def run(self, *args):
+        # Evaluate the condition eagerly, and if false return no nodes right away
+        env = self.state.document.settings.env
+        env.app.builder.tags.add('TRUE')
+        #print(repr(self.arguments[0]))
+        if not env.app.builder.tags.eval_condition(self.arguments[0]):
+            return []
+
+        # Otherwise, do the usual processing
+        nodes = super(EagerOnly, self).run()
+        if len(nodes) == 1:
+            nodes[0]['expr'] = 'TRUE'
+        return nodes
+
+
+def setup(app):
+    directives.register_directive('only', EagerOnly)
diff --git a/docs/sphinx_selective_exclude/modindex_exclude.py b/docs/sphinx_selective_exclude/modindex_exclude.py
new file mode 100644
index 0000000000..18b49cc80f
--- /dev/null
+++ b/docs/sphinx_selective_exclude/modindex_exclude.py
@@ -0,0 +1,75 @@
+#
+# This is a Sphinx documentation tool extension which allows to
+# exclude some Python modules from the generated indexes. Modules
+# are excluded both from "modindex" and "genindex" index tables
+# (in the latter case, all members of a module are exlcuded).
+# To control exclusion, set "modindex_exclude" variable in Sphinx
+# conf.py to the list of modules to exclude. Note: these should be
+# modules (as defined by py:module directive, not just raw filenames).
+# This extension works by monkey-patching Sphinx core, so potentially
+# may not work with untested Sphinx versions. It tested to work with
+# 1.2.2 and 1.4.2
+#
+# Copyright (c) 2016 Paul Sokolovsky
+# Licensed under the terms of BSD license, see LICENSE file.
+#
+import sphinx
+
+
+#org_PythonModuleIndex_generate = None
+org_PyObject_add_target_and_index = None
+org_PyModule_run = None
+
+EXCLUDES = {}
+
+# No longer used, PyModule_run() monkey-patch does all the job
+def PythonModuleIndex_generate(self, docnames=None):
+    docnames = []
+    excludes = self.domain.env.config['modindex_exclude']
+    for modname, (docname, synopsis, platforms, deprecated) in self.domain.data['modules'].items():
+        #print(docname)
+        if modname not in excludes:
+            docnames.append(docname)
+
+    return org_PythonModuleIndex_generate(self, docnames)
+
+
+def PyObject_add_target_and_index(self, name_cls, sig, signode):
+    if hasattr(self.env, "ref_context"):
+        # Sphinx 1.4
+        ref_context = self.env.ref_context
+    else:
+        # Sphinx 1.2
+        ref_context = self.env.temp_data
+    modname = self.options.get(
+        'module', ref_context.get('py:module'))
+    #print("*", modname, name_cls)
+    if modname in self.env.config['modindex_exclude']:
+        return None
+    return org_PyObject_add_target_and_index(self, name_cls, sig, signode)
+
+
+def PyModule_run(self):
+    env = self.state.document.settings.env
+    modname = self.arguments[0].strip()
+    excl = env.config['modindex_exclude']
+    if modname in excl:
+        self.options['noindex'] = True
+        EXCLUDES.setdefault(modname, []).append(env.docname)
+    return org_PyModule_run(self)
+
+
+def setup(app):
+    app.add_config_value('modindex_exclude', [], 'html')
+
+#    global org_PythonModuleIndex_generate
+#    org_PythonModuleIndex_generate = sphinx.domains.python.PythonModuleIndex.generate
+#    sphinx.domains.python.PythonModuleIndex.generate = PythonModuleIndex_generate
+
+    global org_PyObject_add_target_and_index
+    org_PyObject_add_target_and_index = sphinx.domains.python.PyObject.add_target_and_index
+    sphinx.domains.python.PyObject.add_target_and_index = PyObject_add_target_and_index
+
+    global org_PyModule_run
+    org_PyModule_run = sphinx.domains.python.PyModule.run
+    sphinx.domains.python.PyModule.run = PyModule_run
diff --git a/docs/sphinx_selective_exclude/search_auto_exclude.py b/docs/sphinx_selective_exclude/search_auto_exclude.py
new file mode 100644
index 0000000000..b8b326dd2c
--- /dev/null
+++ b/docs/sphinx_selective_exclude/search_auto_exclude.py
@@ -0,0 +1,34 @@
+#
+# This is a Sphinx documentation tool extension which allows to
+# automatically exclude from full-text search index document
+# which are not referenced via toctree::. It's intended to be
+# used with toctrees conditional on only:: directive, with the
+# idea being that if you didn't include it in the ToC, you don't
+# want the docs being findable by search either (for example,
+# because these docs contain information not pertinent to a
+# particular product configuration).
+#
+# This extension depends on "eager_only" extension and won't work
+# without it.
+#
+# Copyright (c) 2016 Paul Sokolovsky
+# Licensed under the terms of BSD license, see LICENSE file.
+#
+import sphinx
+
+
+org_StandaloneHTMLBuilder_index_page = None
+
+
+def StandaloneHTMLBuilder_index_page(self, pagename, doctree, title):
+    if pagename not in self.env.files_to_rebuild:
+        if pagename != self.env.config.master_doc and 'orphan' not in self.env.metadata[pagename]:
+            print("Excluding %s from full-text index because it's not referenced in ToC" % pagename)
+            return
+    return org_StandaloneHTMLBuilder_index_page(self, pagename, doctree, title)
+
+
+def setup(app):
+    global org_StandaloneHTMLBuilder_index_page
+    org_StandaloneHTMLBuilder_index_page = sphinx.builders.html.StandaloneHTMLBuilder.index_page
+    sphinx.builders.html.StandaloneHTMLBuilder.index_page = StandaloneHTMLBuilder_index_page