Extension talk:BookManager/Improve support for book structures

Initial thoughts

This looks great. It's clear you've put a lot of thought into drafting this and I'm impressed with the quality of the writing.

Related to the general idea of books (or collections of pages) is that MediaWiki has, for a long time, worked on a model of subject-space page↔talk page. This works decently for Wikipedia, but for other projects such as Wiktionary, with its Citations namespace, this model doesn't work as well. I'm not sure how much of the work you're doing with the BookManager extension will encounter this issue, but it's something to consider, I think.

The proposal mentions Wikisource and Wikibooks and seems geared toward Wikimedia wikis. Is the intention to have the BookManager extension eventually installed on a Wikimedia wiki (or family of wikis)? If so, I think the sooner the extension can be deployed to a test wiki (on Wikimedia Labs or even test.wikipedia.org), the better. There's also writing an extension for deployment to read and digest, of course.

Regarding the schema changes ("make any necessary changes to the core database..."), do you have an idea of what will be involved here (i.e., which tables you'll want to change or add)? This the one part of the proposal that worries me, as (proposed) schema changes can be difficult for a variety of reasons. --MZMcBride (talk) 21:35, 11 April 2013 (UTC)

Thanks for the feedback!

I'm not quite sure I follow your first point, regarding the Citations namespace and the mainspace/talk namespace model. Both Wikisource and Wikibooks organize their content by collecting a table of contents/index/main landing page for the book on the main namespace page (for example, Haskell or A Jewish State). Then each chapter is located on a subpage of that main page (for example, Haskell/Lists and tuples or A Jewish State/Introduction). The entire work is discussed at the talk page of the main page, and each chapter can be discussed at the subpage talk page. These subpages are all linked together in a somewhat arbitrary way. Wikisource sometimes uses header templates to link chapters together (of which there are several: {{header}}, {{header daily issue}}, {{collective work header}}, etc.) and sometimes uses other ways of navigating (see Science). Sometimes even when the header templates are used, they're not used in the normal way, where the next parameter is the next chapter/volume in a chronological order (see Popular Science Monthly). This project won't affect the way the main pages/subpages are organized; it will, however, replace the content of the main page with a easily-navigable structure that is more standard.

Yes, the intention is to install the BookManager extension on at least Wikisource and Wikibooks. I agree that it should be installed on a test wiki, and will look into how to make that happen.

User:Tpt suggested using JSON to store the book structure and metadata, in a comment on the wikitech-l discussion. I really like this idea, and I think this approach can be used instead of schema changes. Do you agree? GorillaWarfare (talk) 12:31, 12 April 2013 (UTC)

Aims

Hello, thanks for this interesting proposal. However, despite agreeing with its premises, I'm lost right after them and I don't quite understand what you're trying to achieve here.
How did you choose BookManager? I don't know exactly what that extension does because docs are sparse, but the features mentioned there are useless for Wikibooks and Wikisource:

• "Automatic navigation" is already achieved by templates or Proofread Page,
• "Automatic print version" is done by the Collection extension.

The two main reasons we need MediaWiki to know about a "book concept" are the following:

• one-click export of an entire book via Collection extension to PDF, ods, ZIM and ePub format (requests [1] or [2] to which PediaPress preferred [3] or b:User:Pediapress/CollectionParser),
• concept of license and other metadata for the entire book.

The WMF has already worked with PediaPress on the Collection extension (ask User:Tfinc) and volunteers have invested on the Proofread Page extension (Tpt already offered his help): you should probably try and build upon them. This is just my impression and opinion, of course; please advertise your proposal widely on oldwikisource:, other Wikisources, Wikisource-l, textbook-l and the various Wikibooks. Regards, Nemo 08:10, 12 April 2013 (UTC)

Indeed we have in the past. There is no current work being done on this but if I can help guide you in any direction then do let know. Tfinc (talk) 20:53, 16 April 2013 (UTC)

Automatic navigation is approximated, but I would not go so far as to say it's achieved. Various Wikisource projects use different approaches to try to ease navigation, with varying degrees of success (see my above reply to MZMcBride). Because the navigation systems are so different, we really can't rely on every project, say, using the {{header}} template, and even those that do use it will often use it in very different ways.

As for the automatic print version, the Collection extension has not managed this yet. For example, if I wanted to print the Pentagon Papers, I would have to go to each page and manually add it to the book. Many users would expect to be able to click "Create a book" or "Download as PDF" in the sidebar of that main page and export the whole project, but they would be disappointed to see that doing so gives them a 2-page file that only includes the title, the header template, the short description, and some licensing information. They can't even get the full index without manually adding Pentagon-Papers-Index.djvu/9 and Pentagon-Papers-Index.djvu/10 to the book. They can't get the full project without manually adding some 7,000 pages. This is an extreme example because of the particular project's size, but even a 20-page project would be tiring to print because of the need to add each individual page to the book. Tpt's WSexport tool does a better job of handling the book structures, but as of yet it can only export to EPUB2, or several other in-development formats (EPUB3, XHTML, and ODT). It also relies on microformat, so projects like Science that don't use a header template are out of luck.

Although one-click export and license/metadata are very important, they are by no means the only two reasons to create a book structure (see bug 15071 and the bugs it blocks). Matthew Walker has also contacted me to tell me that this project would be useful for an in-browser book rendering project he was working on. GorillaWarfare (talk) 12:59, 12 April 2013 (UTC)

Nemo,

1. The BookManger automatic navigation uses that list and the Proofread Page is done with this. ;)

2.BookManager aims to provide a printable version like this (which for the whole book), and collection does not do it.

Regards.-Raylton P. Sousa (talk) 03:14, 15 April 2013 (UTC)

And I'm pt.wikibooks administrator and assure that it is not useless... But I agree that the docs are sparse - Raylton P. Sousa (talk) 03:23, 15 April 2013 (UTC)

Unfortunately I am unable to get my head fully around the proposal at this point of time. I would say that the Collections extension in its current form is not an efficient tool to generate PDF output of a work with subpages (interepret that as it is a right PITA to use). If there is the means to develop an architecture that will allow means of easy production of other forms than html output, that is a high value feature, especially if new export formats can be easily plugged in. A sequential, minimal click tool is what is required. Extensions that better allows us to simply manage and integrate meta data and navigation are desired, especially where integrating intrawiki and interwiki organisation and data management. — billinghurst sDrewth 00:32, 17 April 2013 (UTC)

I'm a bit confused. If I'm understanding correctly, you're saying that Collections is not efficient, and that a tool to organize multi-page works and associated metadata is high-value because of its potential uses in exportation. From this, it seems that we're in agreement, so what part of the proposal are you not able to get your head around? Or am I misunderstanding your comment? – GorillaWarfare (talk) 12:02, 17 April 2013 (UTC)

wikibooks

A tool or extension that makes it easier to convert a list of wiki pages into a physical paper book -- and otherwise deal with them as a unit -- sounds like a great idea.

I see that you already know about several projects in that general direction. So you probably already know most of the following.

Forgive me for braindumping everything I know about such book-like organizations of wiki pages at Wikibooks:

The vast majority of content wiki pages at Wikibooks are placed in a single place in a single book. However, there are at least 2 exceptions (although I doubt it's worth making things more complicated for everyone just to support two special cases):

• a few standard tables and appendixes are transcluded into many wikibooks: the ASCII table, the Fourier transform table, etc.
• At least one person thought it would be useful to have two different indexes to the same set of wiki pages about C++ ([4]), presenting exactly the same content in two different orders.

There are three five six different ways that all the wiki pages in a single Wikibook are connected:

• Each page of a book has the name of the book (followed by a slash or colon) is prefixed to the page name. A person manually adds that prefix while creating a new page. A person manually changes that prefix when moving a page out of one book and into some other book. See Wikibooks:Wikibooks:Naming_policy. When I go to any page in a book, the Wikibooks software automatically puts a little link at the top of the page that takes me back to the "root page" of the book. Also, the "MediaWiki:Move-subpages" and Extension:AllBooks tools rely on this.
• People manually add {{BookCat}} to each page of the book. That automatically creates "book category page" that lists every page in the book. See Wikibooks:Using Wikibooks/Subjects, Categories, and Classifications.
• People manually list each page of the book on the "root page" index (using link syntax). This lets readers skip all the introductory stuff and jump right to the good stuff :-).
• People manually link up all the other pages of the book (using link syntax), so readers who are reading online can go to the "next" page of the book with a single click. Typically either
  • People sometimes manually edit each page to link to the "previous" and "next" page of the book (in normal reading order), sometimes partially automated by one or another of a huge variety of navigation templates ( Wikibooks:category:navigational templates ). (My understanding is that most people do this manually, because they don't know about the Extension:BookManager ). Or,
  • People sometimes link each page of the book to *every* other page of the book, using book-specific templates like Wikibooks:Template:EuropeanHistoryTOC, manually making a complete list of every page in the book inside the template.
• People manually create yet another list of every page in the book on a sub-page typically named "/Print version" (using transclusion syntax). Transclusion allows us to use tags like <noinclude> to exclude content that is only useful when reading a wiki page by itself online -- such as the above navigation templates, and in some cases the per-page {{reflist}}. It also lets us add things that are not useful when reading a wiki page by itself online -- such as gathering up all the references into a single end-of-book {{reflist}} bibliography. See Wikibooks: Help: Print versions.
  • When a reader views the "/Print version" normally, the Wiki software creates a single long HTML page with all the contents of the book. Some readers prefer reading and scrolling through the entire book that way, rather than reading and scrolling and hitting the "next" button and waiting indefinitely for each page of the book.
  • The "print version" is also a key step towards getting the book on physical paper.
• Occasionally -- such as with Wikibooks:Python Programming -- people manually make yet another list of every page of the book in a sub-page called "/Live print version" (using transclusion syntax), in addition to the "/Print version" mentioned above. (Are you as mystified as I am by this?)

I've been told that Wikipedia:Wikipedia:WikiProject Wikipedia-Books is coordinating teams like Wikipedia:Wikipedia:WikiReader/Cryptography in order to produce books like Wikipedia:Book:Cryptography. I've been told that the Wikipedia approach to creating a book out of wiki articles is totally different than the Wikibooks approach.

I wish you success. --DavidCary (talk) 18:13, 23 May 2013 (UTC)