View source for Blog:Statically hosting a wiki

{{Hatnote|Note: this is a custom-tailored pipeline. You may want to look into [https://github.com/openzim/mwoffliner openzim's ''mwoffliner'' repo].}}

There's a lot of ways you can choose to present your information online, and I think wikis generally have the upper hand on a lot of fronts. I won't go into the benefits—that's what the (currently nonexistent) page [[Blog:Why use a wiki?]] is for—but there's definitely downsides towards trying to use a wiki, majorly with hosting. Given your scope, you could make use out of an existing wikifarm, but generally the more broader it gets (say, with an entire wiki dedicated to any and all of your work) the farms willing to host dwindles down and you could get stuck relying on something like Fandom. You could also choose not to want a wikifarm because you don't expect/want collaborators and wish to keep your content protected, but then you need to deal with hosting the website yourself. In any case, MediaWiki is a dynamic software and unfortunately is extremely expensive, [https://weirdgloop.org/blog/clankers and the majority of internet traffic being greedy scrapers definitely doesn't help].<ref group="note">To summarize, bots really like to scrape every single page they can find—and I mean <em>EVERY</em> page. This is fine if they only scrape what the readers will usually see, but if a greedy or stupid bot (aka almost all of them) stumbles onto more expensive pages like, say, [https://weirdgloop.org/images/posts/clankers_logs.png the most recent changes which are relevant to page whatever with a limit of N days starting from date XYZ excluding minor edits and at max M results], and any of the other billions of combinations of parameters sent to special pages, then the software has to spend extra effort generating all of those pages. Who knows what they're doing this for (my bets are on AI), but it's rampant and horrible.</ref> Oh, also websites cost money, believe it or not? That's probably why you are reading this.

In contrast to all of this, static websites are very easy to host and set up at a cheap price, and there are already many static website hosts like GitHub Pages and Neocities. The obvious solution would be to take a MediaWiki site and turn it into a static snapshot that the service can easily host. This process is actually really easy, but it does have a few drawbacks.

This guide is based on my own experience using a 2018 Mac mini on Sonoma 14.8.7, though I think this should generally be applicable to the common operating systems. Sorry if you are a mobile user...! This blog page won't be static and I'll likely regularly go back and update it; my [https://aadenboy.github.io/wiki/inbox inbox] and the Issues page for this wiki's repo are available if you know of any changes that can be made.

== Setting up the wiki ==
Obviously, the first step is to actually get the wiki set up. This isn't relatively hard, as you just need a way to run SQL and PHP on your system. While you don't need a beefy PC, it does help to have a higher-end device as, again, <em>MediaWiki is expensive</em>. I chose to use MAMP for this, but this is the part where I don't really remember what I did and you're on your own. MediaWiki does have [[mw:Local development quickstart]] and [[mw:Manual:Installation requirements]], though I'm not sure if that's what I used. Anyways, once it's running, MediaWiki should give you a quick setup page where you decide the properties of your wiki, and it should be relatively simple from there.

After the setup process, your wiki should now have an automatically generated "LocalSettings.php" file which contains the settings for your wiki. Most important to you will likely be extensions and namespaces only.

<syntaxhighlight lang="php">
# Important for later!
$wgGroupPermissions["*"]["edit"] = false;

# Extensions that I use for this wiki
wfLoadExtension( 'Cite' );                  # Used for citations.
wfLoadExtension( 'CategoryTree' );          # Shows a tree for each category, if they have subcategories.
wfLoadExtension( 'Gadgets' );               # Allows the use of gadgets.
wfLoadExtension( 'Math' );                  # Renders math equations.
wfLoadExtension( 'TemplateStyles' );        # Allows CSS to be defined in Template namespace.
wfLoadExtension( 'WikiEditor' );            # Provides the toolbar used when editing by source.
wfLoadExtension( 'CodeEditor' );            # Provides a more proper IDE for code pages.
wfLoadExtension( 'TabberNeue' );            # Allows the use of tabs in pages. Use this instead of Tabber.
wfLoadExtension( 'Scribunto' );             # Allows the use of Lua and the Module namespace.
wfLoadExtension( 'Purge' );                 # Adds a purge tab for all pages if you need to refresh the cache.
wfLoadExtension( 'DynamicPageList4' );      # Allows a way to quickly query pages based on their properties and contents.
wfLoadExtension( 'SyntaxHighlight_GeSHi' ); # Allows syntax highlighting code blocks.
wfLoadExtension( 'PortableInfobox' );       # Provides the infoboxes used by Fandom.
wfLoadExtension( 'CodeMirror' );            # Primarily allows syntax highlighting wikitext when editing source.
wfLoadExtension( 'TemplateData' );          # Provides data for templates.
wfLoadExtension( 'TemplateWizard' );        # Allows inserting templates in the source editor via a pop-up.
wfLoadExtension( 'ParserFunctions' );       # Additional parser functions.
wfLoadExtension( 'CreateRedirect' );        # Allows quickly creating redirects to pages.
wfLoadExtension( 'SimpleBatchUpload' );     # Allows quickly uploading many files at once.
wfLoadExtension( 'Interwiki' );             # Provides an interface for managing interwiki links.

# Blog namespace, as an example
define("NS_BLOG", 3500);
define("NS_BLOG_TALK", 3501);
$wgExtraNamespaces[NS_BLOG] = "Blog";
$wgExtraNamespaces[NS_BLOG_TALK] = "Blog_talk";
</syntaxhighlight>

== Editing and styling ==
The neat thing about it being your own private wiki is that you can edit and format it however you like! It does help to have some semblance and keep a mental note of the style of the wiki, but it's entirely up to you what you do and don't include. This is also the only drawback, as unless you find a way to allow others to edit prior to publishing, you're the only contributor. I highly recommend learning how to create templates and modules to make page creation much less time consuming. Also take advantage of any data querying you can get; DPL4 and Bucket are my go-to recommends.

For styling your wiki, you'll only need to edit [[MediaWiki:Common.css]] and [[MediaWiki:Common.js]] at the very least, but it helps to learn what else can be customized and change their respective MediaWiki namespace pages too. If you're unsure, you can append "?uselang=qqx" to any page to see which placeholders are used. You may like [[MediaWiki:Sidebar]] and [[MediaWiki:Tagline]]. Now, of course, it's very easy to go in the wrong direction when formatting and styling a wiki (though take my word however you like), so I'll write a blog page on that later.

== Publishing ==
Now the fun part: publishing! It's actually a rather simple process. If you use Mac or Linux, you should already have the <code>wget</code> command available, though you can still install it if you use Windows. All we're going to use it for is to visit each webpage we want available and save it as an HTML file (alongside any resources). The simplest way to do this is to simply start at one page and crawl as much as physically possible, which works if you really want it to be exactly as seen, but generally isn't the best result and takes a long time.<ref group="note">Ironically, <em>we're</em> now the nasty web scrapers in a way.</ref> For example, all of the pages are located at <code>/wiki/index.php/</code> rather than just <code>/wiki/</code>.

To solve this, I use <code>wget</code> through a Lua script. The script itself is relatively small, only taking a list of pages to scrape through. You can control which pages you want accessible or updated this way, the former of which can be retrieved via [[Special:AllPages]]. It does still take some time to query each page and the prerequisites, but there's not much of a way around that. As a benefit, the script is able to parse the page and modify it to work statically, such as by removing the account buttons and changing links. It also will create an additional page suffixed "?action=edit" which simply reveals the source of the page (assuming you disabled anonymous editing). I've only tested this script with Vector 2010 (the theme this wiki uses), so I don't know how it fares with other skins. Use responsibly, I guess!

<syntaxhighlight lang="lua">
-- https://gist.github.com/liukun/f9ce7d6d14fa45fe9b924a3eed5c3d99
function char_to_hex(c)
    return string.format("%%%02X", string.byte(c))
end
function urlencode(url)
    if url == nil then
        return
    end
    url = url:gsub("\n", "\r\n")
    url = url:gsub("([^%w :_%%%-%.~=])", char_to_hex)
    url = url:gsub(" ", "+")
    return url
end

local repo = "https://github.com/aadenboy/wiki/commits/main/"
function fix(title)
    local safetitle = title:gsub("[%(%)%^%$%%%.%?%+%-]", "%%%0")
    local urltitle = urlencode(title):gsub("[%(%)%^%$%%%.%?%+%-]", "%%%0")
    local urltitle2 = urlencode(title):gsub("%%", "%%%%")
    print(title, safetitle, urltitle, urltitle2)
    local file = io.open("localhost:8888/mediawiki/index.php/"..title..".html", "r")
    local page = file:read("*a")
    file:close()

    page = page:gsub('%s*<nav id="p%-personal".-</nav>', '')
            :gsub('%s*<nav id="p%-cactions".-</nav>', '')
            :gsub('%s*<nav id="p%-search".-</nav>', '')
            :gsub('%s*<nav id="p%-tb".-</nav>', '')
            :gsub('%s*<nav id="p%-namespaces".-</nav>', '')
            :gsub('%s*<link rel="search"[^>]+>', '')
            :gsub('%s*<link rel="EditURI"[^>]+>', '')
            :gsub('%s*<link rel="alternate"[^>]+>', '')
            :gsub('<textarea readonly=""', '<textarea')
            :gsub('%.%/wiki/', '/wiki/')
            :gsub('%/wiki/', '/wiki/')
            :gsub('href="/wiki/'..urltitle, 'href="'..urltitle2)
            :gsub('href="'..urltitle..'%.html', 'href="'..urltitle2)
            :gsub('href="'..urltitle..'#', 'href="#')
            :gsub('http://localhost:8888/mediawiki/index%.php/', '/wiki/')
            :gsub('http://localhost:8888/mediawiki/index%.php/([^"]-)%.html', '/wiki/%1')
            :gsub('http://localhost:8888/mediawiki/index%.php%?title=', '/wiki/')
            :gsub('&amp;action=edit&amp;redlink=1', '')
            :gsub('<li id="ca%-view"[^\n]+', '<li id="ca-view" class="selected mw-list-item"><a href="'..urltitle2..'"><span>Read</span></a></li><li id="ca-viewsource" class="mw-list-item"><a href="'..urltitle2..'%%3Faction=edit" title="View page source [e]" accesskey="e"><span>View source</span></a></li><li id="ca-history" class="mw-list-item"><a href="'..repo..urltitle2..'.html" title="Past revisions of this page [h]" accesskey="h"><span>View history</span></a></li>')
            :gsub('%s*<a class="mw%-jump%-link" href="#searchInput">Jump to search</a>', '')

    local out = io.open("localhost:8888/mediawiki/"..title..".html", "w")
    out:write(page)
    out:close()
end

-- Remember that MediaWiki pages use underscores and (unless you set it otherwise) starts with a capital.
local pagesstr = ([[
Pages go here
Lorem ipsum
File:Dolor sit amet.png
]]):gsub(" ", "_")

local pages = {}
for page in pagesstr:gmatch("[^\n]+") do
    local code = os.execute('wget -E -H -k -K -p --content-on-error "/wiki/'..urlencode(page)..'"')
    fix(page)
    local code = os.execute('wget -E -H -k -K -p --content-on-error "/wiki/'..urlencode(page)..'?action=edit"')
    fix(page.."?action=edit")
end
</syntaxhighlight>

There are likely other bugs I will need to fix sooner, but this works fairly well on my end.

== Contributions ==
While I did say that you're alone when it comes to editing your wiki, you can still allow other people to suggest edits for you. How you go about this is entirely your own preference, but I allow edit suggestions through GitHub Issues, and set up several forms that contributors can fill out. https://github.com/aadenboy/wiki/tree/main/.github/ISSUE_TEMPLATE

{{FootnoteBar}}
<references group="note" />