Coder’s Digest

Abhishek Pandey

Convert GitHub Wiki to Static Site with themes

Aug 7, 2015 · Comments
tipscode githubwikipelican

I recently wanted to setup a wiki so that I could convert it into a static html site with a proper themes. What could be a possible use case for such a requirement:

Anyways, I went about the whole process and decided to jot down everything. Here I am taking D3 Wiki as an example which I will be converting into a static site. Let’s begin.

D3 Wiki using pelican

D3 Wiki using pelican

Setup and requirements

What do we need to get started?

Virtual Environment with pelican

Setup the virtual environment

$ virtualenv ve_blog
$ source ve_blog/bin/activate

Install pelican

$ pip install pelican

Pelican Quickstart

Setup pelican using pelican-quickstart so that all files are setup correctly for creating a static site.

$ pelican-quickstart

Welcome to pelican-quickstart v3.6.3.

This script will help you create a new Pelican-based website.

Please answer the following questions so this script can generate the files
needed by Pelican.

    
> Where do you want to create your new web site? [.] 
> What will be the title of this web site? D3 WIKI
> Who will be the author of this web site? abhi1010
> What will be the default language of this web site? [en] 
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n) n
> Do you want to enable article pagination? (Y/n) Y
> How many articles per page do you want? [10] 
> What is your time zone? [Europe/Paris] Asia/Singapore
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n) Y
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n) Y
> Do you want to upload your website using FTP? (y/N) N
> Do you want to upload your website using SSH? (y/N) N
> Do you want to upload your website using Dropbox? (y/N) N
> Do you want to upload your website using S3? (y/N) N
> Do you want to upload your website using Rackspace Cloud Files? (y/N) N
> Do you want to upload your website using GitHub Pages? (y/N) N
Done. Your new project is available at /Users/apandey/code/githubs/d3wiki

Get the wiki

$ git clone https://github.com/mbostock/d3.wiki.git

Cloning into 'd3.wiki'...
remote: Counting objects: 12026, done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 12026 (delta 607), reused 552 (delta 552), pack-reused 11407
Receiving objects: 100% (12026/12026), 9.92 MiB | 1.49 MiB/s, done.
Resolving deltas: 100% (7595/7595), done.
Checking connectivity... done.

Setting the wiki as content for pelican

$ rmdir content
$ ln -s dr.wiki content

Why simple pelican won’t work and what to do

If you tried to simply call pelican command to build the static site, you will notice a lot of errors like:

$ fab build

RROR: Skipping ./请求.md: could not find information about 'NameError: title'
ERROR: Skipping ./过渡.md: could not find information about 'NameError: title'
ERROR: Skipping ./选择器.md: could not find information about 'NameError: title'
ERROR: Skipping ./选择集.md: could not find information about 'NameError: title'
Done: Processed 0 articles, 0 drafts, 0 pages and 0 hidden pages in 3.47 seconds.

The problem is that pelican expects some variables to be defined in each markdown file before it can build the static file. Some of the variables are:

You may add your own ones as well that you want. However, for our initial purposes, we will keep it simple and just try to add these.

Next, how do we achieve this automation? fab is our answer.

Let’s write a function in python that will modify the markdown files and update them to add Title, Slug, Date

We will edit fabfile.py and add a new function create_wiki:

def create_wiki():
    files = []
    # Find all markdown files in content folder 
    for f in os.walk('./content/'):
        fpath = lambda x: os.path.join(f[0], x)
        for file in f[2]:
            fullpath = fpath(file)
            # print('f = {}'.format(fullpath))
            files.append(fullpath)
    filtered = [f for f in files if f.endswith('.md')]
    for file in filtered:
        with open(file, 'r+') as f:
            content = f.read()
            f.seek(0, 0)
            base = os.path.basename(file).replace('.md', '') 
            lines = ['Title: {}'.format(base.replace('-', ' ')),
                    'Slug: {}'.format(base),
                    'Date: 2015-08-07T14:59:18-04:00',
                    '', '']
            line = '\n'.join(lines)
            # Add the lines to the file
            f.write(line + '\n' + content)
        print(file)
    
    # build and serve the website
    build()
    serve()

Now you can call this function easily:

fab create_wiki

The website can now be viewed at http://localhost:8000

What happened to the menu?

There is a minor issue here though, you will notice that the menu is not available - it is all empty. It is an easy addition. We will need to add some lines to publishconf.py to say what the menu is gonna be.

For my example, I have chosen to show up the following for D3:

Choosing themes

By default, pelican uses its own theme for the static site, but theme can be modified. Let’s choose pelican bootstrap3 for our example here:

git clone https://github.com/DandyDev/pelican-bootstrap3.git

Now, add the full path to the theme at the end of the publishconf.py file:

THEME = "/Users/apandey/code/githubs/pelican_coders/all_themes/pelican-bootstrap3"

Finally, build your site again and serve:

fab build
fab serve
Pelican Bootstrap3 theme

Pelican Bootstrap3 theme

Get all this code in github repo

I realize there maybe a few things going on here. You can get this whole setup as a project from my github repo

You will find all this code and setup so that your life is easier. Just start with d3 wiki along with virtual environment and you will be fine.

comments powered by Disqus