Free book catalogs

Search stays one place, modelled on how Spotify combines browsing and search. There is no separate Discover place, because two places that both find books, one for the library and one for free books, would be confusing.

On desktop and tablet this makes Search a place in the sidebar with its own screen, rather than the palette over the window that RFD 12 describes for the first release. Phones keep the full-screen search they already have. Search over book text and notes keeps the scopes and grouping of RFD 16.

Every free-book result and tile opens a book page: cover, title, author with life dates, first publication, language, summary, subjects and reading ease where the source has them. One work can have several editions, for example a Standard Ebooks edition and two Gutenberg files, or one Aozora work in several orthographies. The page lists them with their sizes, the best edition selected first. Add to library downloads the chosen file and adds it like any other book. The page shows the edition’s rights line and credits its source.

The Aozora tiles open an author index by kana rows (あかさたな…), which only Aozora’s kana readings make possible, with filters for orthography and NDC class. Aozora works have no covers, so Shōsai draws a cover from the title and author in the book face.

Adding an Aozora work downloads its XHTML, converts it from Shift_JIS, and wraps it in an EPUB 3 with vertical writing, the original ruby, and Aozora’s colophon (底本, 入力, 校正) kept as the license asks. From then on it is an ordinary EPUB in the library and uses the vertical text, ruby and emphasis-dot support Shōsai already has.

A scheduled job builds one merged catalog from the sources' own catalog files and publishes it as a static site on Cloudflare Pages, either on the Shōsai website or as a separate project. It runs weekly, matching Gutenberg’s weekly catalog file, and can run daily because a build takes seconds. It publishes:

The app checks the manifest at most once a day while Search is in use, downloads the catalog only when it changes, verifies the checksum, builds a trigram full-text index on the device (about 3 seconds, needed for Japanese substring search) and swaps the new catalog in atomically. The catalog lives in its own database file, separate from the library store, so replacing it never touches the person’s books or notes.

Covers and book files load from each source when they are needed. Browsing downloads metadata and small cover images only.

Publishing the index rather than calling each source live gives one search across sources with merged editions, filters and offline use, and keeps the app’s traffic to each source down to cover images and the books people add. Building it ourselves, rather than on each device, keeps the app from downloading and parsing the sources' raw catalog files, lets us merge editions and fix data in one place, and lets us add sources without an app update.

Each book page shows its source’s rights line, such as "Public domain in the USA", followed by "If you live elsewhere, check your local laws." Shōsai does not block books by region. Every edition credits its source by name with a link to it; Standard Ebooks editions also link to the page where readers can support the project. The app’s About screen and the published index credit Aozora Bunko’s catalog data under CC BY 4.0.

The catalog leaves out Aozora works that are still in copyright, which Aozora shares for private reading only, and Gutenberg’s audio books.

The catalog needs no account and sends nothing about the person. The app fetches the manifest and catalog from our site, and covers and book files from the sources. Requests identify Shōsai in their User-Agent, as the sources ask. Searches run on the device.

A Discover place beside Library and Search, with its own shelf of free books and its own search, was mocked up first. It was rejected because two places that both find books, differing only in where the books come from, are confusing.

The app could call each source directly with no index and nothing to host. Gutenberg’s feed lists arrive without covers or summaries, there is no cross-source search or merging, Aozora has nothing to call, and Gutenberg’s XML feed ends in 2027.

The app could download Gutenberg’s and Aozora’s catalog files and build the index itself. That avoids hosting, but every device repeats the parsing and merging, data fixes need an app update, and some sources could not be added this way at all.

Large, but the files are OCR’d scans and the language data is noisy. Open Library may help later to match editions and fill in covers and subjects.

Adding any OPDS catalog, such as a Calibre server or a library, is a power-user feature that does not help most readers find a book. It can come later on top of the same book page.