Forbajato Techblog2016-03-18T09:56:42+00:00http://forbajato.github.ioThomas Hicksshuangyisheng@gmail.comPoor man's VPN2014-08-25T00:00:00+00:00http://forbajato.github.io/tutorials/2014/08/25/poor-mans-vpn
<h1 id="why-would-you-want-a-tunnel-anyway">Why would you want a tunnel anyway?</h1>
<p>Tunnels are great, they are mysterious, they can be a little scary. Of course we are not talking about train tunnels or roadway tunnels but ssh tunnels here. In our kind of tunnels the goal is to send network packets through the tunnel from your computer to a website and then get the responses back. VPNs can accomplish this nicely with really high level encryption, ssh can do something similar but with slightly less encryption.</p>
<h1 id="materials">Materials</h1>
<p>To do this you will need a Linux computer with ssh capbabilities and a server somewhere (I use a Linode server for this). We will set up the tunnel to the server and then set Firefox to point to the local tunnel as a proxy. Let’s get going.</p>
<h1 id="start-the-ssh-tunnel">Start the ssh tunnel</h1>
<p>The basics here are easy, say your external server is at ip address <code class="highlighter-rouge">11.11.11.11</code> and you have a user account on that server with username <code class="highlighter-rouge">me</code>. Now you want to start a tunnel. All you need to do is start an ssh session with your server and bind that session to a local port:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>ssh -D 12345 me@11.11.11.11
</code></pre>
</div>
<p>Of course the <code class="highlighter-rouge">ssh</code> starts the ssh connection. The <code class="highlighter-rouge">-D</code> is really cool - it binds the connection on the remote server to the numbered port listed next (in this case “12345”). After you do this, when you try to connect to localhost on port 12345 all the traffic will be sent down the tunnel to the remote server. Since you are doing this over ssh the tunnel is secure.</p>
<h1 id="point-firefox-at-the-newly-created-proxy-server">Point Firefox at the newly created proxy server</h1>
<p>Firefox is happy to send its traffic anywhere you would like. Normally it just follows the rules on your system or router but this is a special case where you want to send that traffic out the tunnel. To do that you have to make some changes in Firefox.</p>
<p>Open the Preferences dialog window (Edit>Preferences or click “Preferences” from the menu button usually in the top right of the Firefox window). Choose the “Advanced” tab and click on the “Network” tab. The dialog you see will look like the photo below.</p>
<p><img src="/assets/PreferencesAdvancedNetwork.png" alt="Network Settings Tab" /></p>
<p>Click the “Settings” button. The new dialog will look like the screenshot below:</p>
<p><img src="/assets/PreferencesAdvancedNetworkSettings.png" alt="Network Settings Dialog" /></p>
<p>Select the “Manual proxy configuration” radio button, the boxes under it will be come active. In the ‘SOCKS Host” box put <code class="highlighter-rouge">127.0.0.1</code> and make the corresponding port <code class="highlighter-rouge">12345</code>. Make sure the <code class="highlighter-rouge">SOCKS v5</code> radio button is selected.</p>
<p>For a lot of things this may work. The problem is you can run into DNS settings on some networks (or in some countries) where the DNS servers are hijacked and return gibberish when you try to connect to a disallowed service. To make sure all the DNS traffic goes through the tunnel you need one more step.</p>
<h1 id="edit-aboutconfig-preferences-que-lightening-strike-and-ominous-music">Edit about:config preferences (que lightening strike and ominous music)</h1>
<p>Firefox knows that editing the about:config preferences can do some major damage so it will warn you that this will void your warranty (you know the warranty you signed when you started using open source software? You mean you didn’t sign it?!? I guess you can forge ahead risk taker!). Click the <code class="highlighter-rouge">I'll be careful, I promise</code> button.</p>
<p>The page that loads is a confusing array of options for controlling how Firefox works - remember all those options with your former closed source software? No? Probably because they weren’t there. With great power though, comes great responsibility, seriously, be careful with this.</p>
<p>In the search bar at the top enter “dns” - you should get a much shorter list of options. The one you are looking for is <code class="highlighter-rouge">network.proxy.socks_remote_dns</code>, if you haven’t done anything crazy to your system in the past it should have a Status of <code class="highlighter-rouge">default</code>, a Type of <code class="highlighter-rouge">boolean</code>, and a Value of <code class="highlighter-rouge">false</code>. All we need to do is change the Value to <code class="highlighter-rouge">True</code> - do this by double clicking where it says <code class="highlighter-rouge">False</code> - the line will turn bold and the Value will change to <code class="highlighter-rouge">True</code>.</p>
<p>Now, supposedly you don’t have to restart Firefox after you make these changes but I have better luck if I do a restart. Once you restart Firefox you will be surfing through the tunnel (or is the correct term ‘pipe’ when you are talking surfing?).</p>
<p>When you are ready to close the tunnel simply disconnect from the ssh connection and you are back to the original configuration. You will have to move your Firefox back to “No Proxy” in the Preferences>Advanced>Network>Settings screen once you close the tunnel. This can be automated by setting up a Firefox profile with the required proxy settings that can be run when you are using the tunnel, then run regular Firefox (i.e. no proxy settings) when you want to use the normal connection.</p>
<p>This is basically just the steps I took to do this while following a far more complete tutorial <a href="https://www.linode.com/docs/networking/ssh/setting-up-an-ssh-tunnel-with-your-linode-for-safe-browsing">here</a>.</p>
Encrypting containers in Linux2014-06-26T00:00:00+00:00http://forbajato.github.io/blog/2014/06/26/encrypting-containers-in-linux
<p>#Rationale</p>
<p>You may have heard that Truecrypti is dead. There is lots of speculation as to why and I won’t go into that, suffice to say it is no longer considered safe to use truecrypt to encrypt your hard drive or any device attached to your machine. What I always loved about truecrypt was its ability to encrypt containers that could then be mounted on multiple systems. Say you have a USB drive that you want encrypted - easy. Say you just want part of it encrypted - also easy. It turns out, you can do the same thing with encrypted containers in Linux - have been able to do so for years! Let’s get started …</p>
<p>#Necessary packages</p>
<p>You may already have this (do <code class="highlighter-rouge">which cryptsetup</code> to find out). If you don’t have it, cryptsetup is available in most repos. On Debian based systems simple apt-get install:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo apt-get install cryptosetup
</code></pre>
</div>
<p>This will allow you to create and manipulate encrypted containers easily. The website for cryptsetup is <a href="https://code.google.com/p/cryptsetup">here</a>.</p>
<p>#Make a container</p>
<p>Linux comes with a wonderful little utility called <code class="highlighter-rouge">dd</code> - it is definitely the kind of thing about which Uncle Ben would advice Peter however - we are talking great power here. This utility is sometimes referred to as the “disk destroyer” so be careful and know the switches and options you are using!</p>
<p>For creating a container we will make a file and write all zeros to it. <code class="highlighter-rouge">dd</code> helps us with this by taking an input file parameter (<code class="highlighter-rouge">if=</code>), the zeros come from the /dev/zero file. You next need to tell <code class="highlighter-rouge">dd</code> where to put the file and what to call it, this is the output file parameter (<code class="highlighter-rouge">of=</code>), you can tell it wherever you want it to go. Keep in mind that if you run <code class="highlighter-rouge">dd</code> as a normal user not only will you be unlikely to totally hose your system but the resulting container will have your user permissions - a handy feature!</p>
<div class="highlighter-rouge"><pre class="highlight"><code>dd if=/dev/zero of=<path to container>/one.bin count=2000k
</code></pre>
</div>
<p>The above will create a 1 gigabyte file with all zeros and call it <code class="highlighter-rouge">one.bin</code>.</p>
<p>#Connecting to loopback device</p>
<p>After you make your container you will need to connect it to a loopback device before you can encrypt it. You can find the loopback devices in your system with the following:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>ls -liha /dev/loop*
</code></pre>
</div>
<p>This will list a bunch of files that can be used to connect to your container. The <code class="highlighter-rouge">losetup</code> command connects loopback files to containers (you may have used it to mount iso files to test distros in the past? What you aren’t a distro hopper?).</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo losetup /dev/loop1 <path to container>/one.bin
</code></pre>
</div>
<p>We attached to loop1 but could have used any of them (my system has 7 possibilities).</p>
<p>#Partitioning the container</p>
<p>Before we can format the container we need to put a partition table on it. For that there are a number of tools (gparted if you like GUI tools, fdisk if you are a terminal hound). I will leave figuring out how to put the partition on the loopback device to you but suffice to say if you use fdisk (like the command below) accepting the defaults works fine.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo fdisk /dev/loop1
</code></pre>
</div>
<p>#Encrypting the partition</p>
<p>Now we finally get to use cryptsetup to encrypt the container (even before there is a filesystem on it!).</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo cryptsetup -v -y luksFormat /dev/loop1
</code></pre>
</div>
<p>The <code class="highlighter-rouge">-v</code> parameter means “give me verbose output” and can be replaced with <code class="highlighter-rouge">--verbose</code>. The <code class="highlighter-rouge">-y</code> parameter means “verify my passphrase, get it twice” and can be replaced with <code class="highlighter-rouge">--verify-passphrase</code>. After the parameters is <code class="highlighter-rouge">luksFormat</code> - basically you are initializing the LUKS partition. Be warned - if you run this command on a container that already has a LUKS partition the old one will be completely replaced. Apparently there is a way to retrieve it (something about header backups) but you are on your own there!</p>
<p>##Opening the device</p>
<p>Now we have to open the device - this is done by opening the LUKS partition and telling the file system to map it to a particular spot. We need it in a spot in the filesystem so we can later mount it for access.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo cryptsetup luksOpen /dev/loop1 cryptStuff
</code></pre>
</div>
<p>This will set up a file in <code class="highlighter-rouge">/dev/mapper</code> that is called <code class="highlighter-rouge">cryptStuff</code>. As you likely know, things in the /dev tree are easily mounted into the filesystem (as long as they have a filesystem on them!). So let’s put a filesystem on this bad boy.</p>
<p>#Formatting the encrypted container</p>
<p>You can’t write to a non-formated container so let’s fix that. Interestingly, you are supposed to be able to use mkfs as a regular user but I can’t on my system. If I make the filesystem as the root user (using <code class="highlighter-rouge">sudo</code>) then it will be owned by root and be in the root group. This may be inconvenient at times so you can specify who the root owner should be by specifying a UID and GID when you make the filesystem. That is in the extended (<code class="highlighter-rouge">-E</code>) options for mkfs.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo mkfs.ext4 -j -E root_owner=<UID>:<GID> /dev/mapper/cryptStuff
</code></pre>
</div>
<p>If you don’t know your user UID and GID you can find it in the passwd file like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>less /etc/passwd | grep <username>
</code></pre>
</div>
<p>The output will have something like this (assuming your username is <code class="highlighter-rouge">dude</code> and why would it not be?):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>dude:x:####:####:dude.........
</code></pre>
</div>
<p>The <code class="highlighter-rouge">####</code>’s in the above are the UID and GID respectively.</p>
<p>#Mount the device</p>
<p>Mounting is a simple matter. You should have a place to mount it:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>mkdir /home/dude/cryptStuff
</code></pre>
</div>
<p>Then just mount it:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo mount /dev/mapper/cryptStuff /home/dude/cryptStuff
</code></pre>
</div>
<p>Now you are ready to use.</p>
<p>#What about shutting it down?</p>
<p>This is no big deal if all you need to do is shut it down when you shut down the computer, everything will be handled automatically. If you want to shut the encrypted container without shutting off the computer just do things in reverse:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo umount /home/dude/cryptStuff
sudo cryptsetup luksClose /dev/mapper/cryptStuff
sudo losetup -d /dev/loop1
</code></pre>
</div>
<p>Nothing to it!</p>
<p>#What about starting it up the next time?</p>
<p>Obviously now the device is made, it has a partition on it and the partition is formated. All you have to do is connect it to a loop device, map it and mount it:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo losetup /dev/loop1 /home/dude/one.bin
sudo cryptsetup luksOpen /dev/loop1 cryptStuff
sudo mount /dev/mapper/cryptStuff /home/dude/cryptStuff
</code></pre>
</div>
<p>And voila!</p>
<p>I leaned heavily on the post <a href="http://www.linux.org/threads/encrypted-containers-without-truecrypt.4478/">here</a> with a few changes and comments to make it clearer to me for future use. If this isn’t clear to you you should check out that post - it is quite good. Alternatively you could take a chance that I could help and leave a comment, I would love to hear from you if this tutorial was helpful or confusing.</p>
Python Puzzle Counting characters in a string2014-04-29T00:00:00+00:00http://forbajato.github.io/blog/2014/04/29/python-puzzle---counting-characters-in-a-string
<p>#The Puzzle</p>
<p>This is a classic beginning python puzzler - if you are given a list of characters, how many times does “a” (or any other given character) occur? The usefulness is obvious if you are thinking about big data sets - say you want to find rarely occurring letters or the most commonly occurring ones. The principle would be the same if you were looking for words or nucleotide base sequences too I suppose. The Checkio puzzle (found <a href="http://www.checkio.org/mission/most-wanted-letter/">here</a>) states you need to find the most commonly occurring letter and report it. If all the letters are the same frequency then report the earliest letter in the alphabet.</p>
<p>#Counting with dictionaries</p>
<p>A pretty common pattern to figure out the frequency of things is to count them in a dictionary. All you have to do is set up an empty dictionary and then accumulate items to it. Whenever a copy of the item (in the dictionary this will become what we call a “key”) is already in the dictionary, increase that item’s value by one. If the item is not yet in the dictionary, add it.</p>
<p>So, first set up an empty dictionary:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>chars = {}
</code></pre>
</div>
<p>Next you will want to loop over a string (we will call ours “text” since “string” is a reserved word in Python) and check to see if each character in the string is already in the dictionary or not:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>for character in text:
chars[character] = chars.get(character,0)+1
</code></pre>
</div>
<p>Pretty simple! Now, what is going on here? We started the <code class="highlighter-rouge">for</code> loop, telling it to loop over all the characters in the string “text”. The <code class="highlighter-rouge">chars.get</code> looks to see if the character is in the dictionary already. If the character is not in the dictionary already then it is added, if it is there then its value is increased by 1. When the <code class="highlighter-rouge">for</code> loop is finished we will have a frequency table of characters in the string.</p>
<p>What if we want to make sure we only count letters (no special characters like spaces or punctuation)? Simple just add:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>if character.isalpha
</code></pre>
</div>
<p>to the <code class="highlighter-rouge">for</code> loop. Want to report only lower case characters? A single character from a string of characters still gets all the string methods available in the standard library. As such you can convert it and to other interesting things as needed. To convert each character into the lower case version before checking it in the dictionary:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>character = character.lower()
</code></pre>
</div>
<p>Now, putting it all together:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>chars = {}
for character in text:
character = character.lower()
if character.isalpha():
chars[character] = chars.get(character,0)+1
</code></pre>
</div>
<p>When that part of the function finishes we have a dictionary (called <code class="highlighter-rouge">chars</code>) that has key:value pairs of each letter occurring in the string as the key and the number of times it occurs as the value.</p>
<p>Now we need to find the key with the highest value. It would be great if we could use something like the <code class="highlighter-rouge">max</code> method you use on iterables but we are using a dictionary. But what if we took just the values in the dictionary, made them an iterable, found the highest one and the looked to see what key went with that value?</p>
<div class="highlighter-rouge"><pre class="highlight"><code>for k,v in chars.iteritems():
if v == max(chars.values()):
return k
</code></pre>
</div>
<p>Nothing to it. The <code class="highlighter-rouge">chars.iteritems()</code> turns our dictionary into an iterable. Then we look at the values in the dictionary, find the highest level one, look up its key and Voila! we have the letter that occurs most frequently in our text.</p>
<p>That seems like a lot of work though. I am sure you can find a better way to do this, maybe exploring other iterables (like sets) and how the <code class="highlighter-rouge">max</code> function works.</p>
Ipython debugging - where have you been all my life?!?2014-03-30T00:00:00+00:00http://forbajato.github.io/blog/2014/03/30/ipython-debugging
<p>Ever write a bit of quick code and then run it in your interpreter only to find it error out? How do you go about debugging? I generally look at the output, wish I understood it better and then start putting in print statements so I can figure out what is going on at various places in the code. What if there was a better way? I had heard about this python debugger (pdb) but had never really understood what it did. Mike Driscoll, of the Mouse vs. Python blog, has done an awesome job explaining the python debugger (you can check it out <a href="http://www.blog.pythonlibrary.org/2014/03/19/pytho-101-an-introduction-to-pythons-debugger/">here</a>). Only problem - I use Ipython so what does this mean to me?</p>
<p>Ipython employs magic commands to do a number of functions, debugging is one of them. For instance, say you have the same silly code from Driscoll’s article:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def doubler(a):
result = a*2
print result
return result
def main():
""""""
for i in range(1,10):
doubler(i)
if __name__ == "__main__":
print my_result
main()
</code></pre>
</div>
<p>OK, I changed the code a bit so that we would throw an exception to demonstrate the %debug magic word. Run the code in the ipython interpreter:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>%run debug_test.py
</code></pre>
</div>
<p>If you do this you will get an error like so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>NameError Traceback (most recent call last)
/usr/lib/python2.7/dist-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
176 else:
177 filename = fname
--> 178 __builtin__.execfile(filename, *where)
/home/forbajato/pdb_learning/debug_test.py in <module>()
13
14 if __name__ == "__main__":
---> 15 print my_result
16 main()
17
NameError: name 'my_result' is not defined
</code></pre>
</div>
<p>This is a trivial example and the error is likely obvious but if it wasn’t you could use the %debug magic word to figure things out:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>None
> /home/forbajato/pdb_learning/debug_test.py(15)<module>()
14 if __name__ == "__main__":
---> 15 print my_result
16 main()
</code></pre>
</div>
<p>Now you have an arrow pointing to the line with the error in it and a line showing the offending line (that is what that <code class="highlighter-rouge">debug_test.py(15)</code> means). The debugger doesn’t reproduce the entire trace of the error - that is helpful to me since I figure the standard library (the part pointed to by the <code class="highlighter-rouge">/usr/lib/pyth...</code> line) is OK and I don’t need to change it much. Now all I have to do is fix that line, in this case we will delete it.</p>
<p>Now when I do a <code class="highlighter-rouge">%run debug_test.py</code> I everything works great. What if I wanted to step through it one step at a time. This could be helpful if the errors are complicated and I need to see the code piece by piece or, alternatively, if the code is someone else’s and I am trying to figure out what it does. Stepping through code line by line can be both useful to debug and educational to study.</p>
<p>To run through the code step by step you can just tell the <code class="highlighter-rouge">%run</code> magic command to help you with that:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>%run -d debug_test.py
</code></pre>
</div>
<p>This output will work through the file line by line, putting an arrow at the line currently under execution and showing it to you. The first line outputs something like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>> /home/forbajato/pdb_learning/debug_test.py(3)<module>()
2
1---> 3 def doubler(a):
4 """"""
</code></pre>
</div>
<p>Now you can see that the first function, doubler, is being called. Go to the next line to execute by entering <code class="highlighter-rouge">n</code> in the <code class="highlighter-rouge">ipdb></code> prompt. All you have to do is work your way through all the steps until you see the total execution.</p>
<p>Other useful commands for the ipython debugger are jump (j) and quit (q). With jump (enter a <code class="highlighter-rouge">j</code> instead of a <code class="highlighter-rouge">n</code> at the <code class="highlighter-rouge">ipdb></code> prompt) allows you to go to a specific line of code to execute next. For instance entering <code class="highlighter-rouge">ipdb>j 15</code> allows you to jump down to the business end of the <code class="highlighter-rouge">if __name__ == "__main__":</code> section and move on from there.</p>
<p>There is lots more to learn about debugging but this should get things started. Have fun and keep learning!</p>
Fixing i3status and window titles2014-01-28T00:00:00+00:00http://forbajato.github.io/blog/2014/01/28/fixing-i3status-and-window-titles
<p>#Everything is configurable</p>
<p>i3 allows you to configure and change pretty much everything. I am still loving using the window manager and have found it can make my work flow faster and much more efficiently. Ran across <a href="http://xpressrazor.wordpress.com/2014/01/27/introduction-to-i3/">this post</a> with some more tips on using and customizing i3.</p>
<p>#i3status</p>
<p>i3status is documented separately from the normal i3 config, check <code class="highlighter-rouge">man i3status</code> for the man page. You can change what items are on the status line through the config file (located at <code class="highlighter-rouge">/etc/i3status.conf</code>). You can change how it appears in the normal i3 conf file though - located at <code class="highlighter-rouge">.i3/conf</code>. I made a few changes to the <code class="highlighter-rouge">bar</code> section of the i3 configuration file:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>bar {
status_command i3status
font pango:Droid Sans, Ioicons, FontAwesome 8
colors {
background #000000
statusline #33aaff
focused_workspace #11aaff #005500
active_workspace #11aaff #005500
urgent_workspace #ffffff #990000
}
}
</code></pre>
</div>
<p>#Even change the window title fonts!</p>
<p>OK, so this isn’t really hidden but up in the <code class="highlighter-rouge">.i3/config</code> there is a section that defines the font for the window titles. Look for the <code class="highlighter-rouge"># Font for window titles</code> section. I changed the font line to give me Verdana font for my window titles:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>font pango:Verdana 10
</code></pre>
</div>
<p>Nothing to it!</p>
Setting up an Nginx server on Raspberry Pi2014-01-20T00:00:00+00:00http://forbajato.github.io/lessons/2014/01/20/setting-up-an-nginx-server-on-raspberry-pi
<p>#Introduction</p>
<p>I don’t know how he knew I would love this but my nephew got me a raspberry pi for Christmas. I have been working on my VPS and want to deploy a webserver on it but am a little reticent to deploy to the Internet without experimenting in a safe environment first. The RPi solves this problem very neatly - just install Raspbian, fire up a package manager to install nginx and off we go!</p>
<p>#Goals</p>
<ol>
<li>Install and configure Raspbian.</li>
<li>Install nginx webserver on RPi.</li>
<li>Set up RPi server to host static pages blog.</li>
<li>Set up local computer to deploy static pages blog to RPi server.</li>
</ol>
<p>#Install and configure Raspbian</p>
<p>Start by downloading the Raspbian distro from <a href="http://www.raspberrypi.org/downloads">here</a>. Page down until you see the Raspbian section, it has a link for both a direct download of the image file and a link for the torrent. I went with the torrent so I could help seed the distro afterwards (gotta be a good netizen!). If you are a total newbie to Linux and Raspberry Pi you may want to go with the NOOBS install - it will enable you to choose which distro to install and even change it later. I know my VPS, which I am trying to model, runs Ubuntu so I don’t need to mess around with the other versions of the RPi operating systems and went straight with the Raspbian.</p>
<p>Get your SD card ready - I used gparted to clear out an 8G SD card, you will need at least a 4G card to install Raspbian and have something workable to run it afterwards.</p>
<p>Flash the SD card with the Raspbian image. This is simple in Linux:</p>
<p><code class="highlighter-rouge">$sudo dd bs=4M if=./raspbian.img of=/dev/location_of_the_sd_card</code></p>
<p>Replace the <code class="highlighter-rouge">./raspbian.img</code> with the path to and name of the image file, replace <code class="highlighter-rouge">location_of_the_sd_card</code> with the spot where the SD card is in your system. In Linux you can find this by plugging in the SD card and then doing <code class="highlighter-rouge">dmesg</code> in the terminal, it will show you where the SD card went in your file system (for me it was <code class="highlighter-rouge">/dev/mmcblk0</code>). This process may take a minute or two. When it is finished you are ready to fire up your RPi.</p>
<p>Insert the newly flashed SD card into the RPi, plug in the video (either using composite to the TV or the HDMI cable), a keyboard and (ideally) an ethernet cable or wireless USB card. Finally plug in the power supply to boot up the RPi.</p>
<p>On first boot Raspbian will run raspi-config, a tool to get things up and running quickly. Your user name will be <code class="highlighter-rouge">pi</code> and the password will be <code class="highlighter-rouge">raspberry</code>, this user has sudo access.</p>
<p>First you will want to expand the filesystem. When you flashed the SD card you only installed a boot and system partition, there may well be lots of SD card that is not being used, you don’t want it going to waste!</p>
<p>Next you can work through the various options on the rasp-config screen - reseting the password (the pi user has admin access so you will not want to leave that user with the default password if this thing is going to be exposed to the Internet), setting up networking, etc. There are even options to set the machine to boot to an X server or directly to Scratch - I will have to come back an play with those sometime. One thing you want to be sure to set up is the SSH server, once that is done you can administer the RPi from any with an ssh client - i.e. you won’t need the mouse, keyboard and monitor (or TV in my case) anymore.</p>
<p>After everything is set up the first thing to do is to update the sources so you can get to installing things.</p>
<p><code class="highlighter-rouge">$sudo apt-get update</code></p>
<p>#Install nginx webserver on RPi.</p>
<p>To use this as a webserver we will need to install one, there are several choices but we are only hosting static pages on this so nginx is a good choice. Nginx seems pretty simple without too much to get it up and running yet hefty enough to handle huge loads without blinking.</p>
<p>First install the server:</p>
<p><code class="highlighter-rouge">$sudo apt-get install nginx</code></p>
<p>If you installed from the Raspbian repos you already have the www-data user and www-data group. To be sure (or if you installed in another way):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$sudo useradd www-data
$sudo groupadd www-data
$sudo usermod -g www-data www-data
</code></pre>
</div>
<p>Now you need to check the nginx configuration files - I made a copy before I started messing with this, always a good idea to have a back up just in case you do something … ill-advised (aka, “stupid”). The first config file to adjust is /etc/nginx/sites-enabled/default. If you are not using IPV6 you will need to comment out (add a <code class="highlighter-rouge">#</code> sign to the beginning of the line) the line that looks like:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>listen [::]:80 default_server ipv6only=on;
</code></pre>
</div>
<p>My file started with both this and the line above it commented out, I uncommented (removed the <code class="highlighter-rouge">#</code>) the line above it <code class="highlighter-rouge">listen 80;</code> so that the server knew where to listen.</p>
<p>Next you will need to set up where you are going to put your website files. I plan to use a subdirectory of the pi user <code class="highlighter-rouge">/srv/web</code> but nginx comes ready to use <code class="highlighter-rouge">/usr/share/nginx/www</code>, I am not sure of the advantage of this approach, will look into it and may end up changing my set up in the future. For now we are just wanting to get things running so you need to make a change to the default file to add the following line:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>root /home/pi/srv/web;
</code></pre>
</div>
<p>You will also want to comment out (put a <code class="highlighter-rouge">#</code> at the beginning) the line that says:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>root /usr/share/nginx/www;
</code></pre>
</div>
<p>Now you need some stuff in /var/www and you need to be sure that directory belongs to the proper user. First make the directory:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$cd /var
$sudo mkdir www
$sudo chown www-data:www-data www/
</code></pre>
</div>
<p>Now you need something in that directory to serve so you can test things out. Fire up a text editor and write some simple HTML, save it as <code class="highlighter-rouge">index.html</code> in the <code class="highlighter-rouge">/var/www</code> directory. Finally start up the nginx server:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$sudo service nginx start
</code></pre>
</div>
<p>If you don’t already know the RPi’s IP address issue this at the command line:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ifconfig
</code></pre>
</div>
<p>In the block eth0 (if you have an ethernet cable connected to the RPi) or the wlan0 (if your RPi is using wireless) will have an IP address, you will use that to test the server. From another computer on your network simply point the browser at the IP address of your RPi and you should see the webpage you just created. If so, so far so good!</p>
<p>Much thanks to the tutorial <a href="http://en.joscandreu.com/blog/install-nginx-on-raspberry-pi/">here</a> which is a great resource for doing this little project. Also the <a href="https://library.linode.com/web-servers/nginx/installation/debian-6-squeeze#sph_installing-nginx-from-debian-packages">Linode</a> documentation was invaluable.</p>
<p>#Deploying the pages</p>
<p>Now you have your server set up, all it needs is some content. I fiddled around with using git to publish (like I do to post to github) but it turns out to be a bit complicated. Turns out good ‘ole rsync can send files over to the RPi just fine with a simple</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$rsync -avx "_site/" pi@rpi_ip_address:/home/pi/srv/web/
</code></pre>
</div>
<p>You can even automated it with rake by setting a new rake task. All you have to do is edit your Rakefile (I have one for making new posts, pages, etc. in my git repo directory where the source files for the static blog live) to include the following (doesn’t seem to matter where you put it in the file):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>desc "rsync the contents of _site directory to the raspberry pi"
task :syncpi do
puts '* Publishing files to RaspberryPi'
puts `rsync -avz "_site/" pi@rpi_ip_address:/home/pi/srv/web/`
end
</code></pre>
</div>
<p>Those are backticks around the rsync command. Now to post your blog all you have to do is fire up Jekyll’s server, let it make the <code class="highlighter-rouge">_site/</code> directory, then do a <code class="highlighter-rouge">rake syncpi</code> to send it to the RPi for hosting.</p>
<p>Thanks to <a href="http://arjanvandergaag.nl/blog/publishing-a-jekyll-website-to-a-server.html">this blogger</a> for help with the rsync and rake file tips on posting.</p>
Favorite Firefox plugins2014-01-13T00:00:00+00:00http://forbajato.github.io/blog/2014/01/13/favorite-firefox-plugins
<p>#Introduction</p>
<p>Open source software affords the opportunity to use your applications for all they are worth. Some projects even allow the extension of an application’s functions by writing plugins (or “extensions”). Firefox is one of those applications. I haven’t used Safari ever and it has been years since I used Internet Explorer so I don’t know if they offer this but the ability to really take control of what my browser is doing keeps me coming back to Firefox (and occasionally Chrome). Below is my list of favorite plugins with comments on what they do and how I use them.</p>
<p>##Adblock Plus</p>
<p>Let’s be honest, the Internet is built and paid for to a large extent by advertising. That being said, I still would rather not look at ads if possible. <a href="http://adblockplus.org/en/">Adblock Plus</a> is a great plugin to eliminate lots of advertising from sites you visit. I had to go back and disable this plugin to view a few ads just to remind myself of what I am missing! Very effective, very easy to use an install.</p>
<p>##Reddit Enhancement Suite</p>
<p>If you don’t use Reddit you may want to ignore this one. For those that use Reddit this is a really sweet little plugin. With <a href="https://addons.mozilla.org/en-US/firefox/addon/reddit-enhancement-suite/?src=api">RES</a> you can alter the interface of Reddit, follow conversation threads easier and a host of other things. I use this most for editing the subreddit menu at the top of the screen and hitting the <code class="highlighter-rouge">[l+c]</code> button to open the linked item and reddit comments. It also includes a lot of keyboard navigation enhancements but I have that covered with Pentadactyl (see below).</p>
<p>##Scrapbook</p>
<p>The first plugin or extension I add to any install of Firefox. <a href="http://amb.vis.ne.jp/mozilla/scrapbook/">Scrapbook</a> captures a webpage with all its links, pictures, etc. intact. I have used this to prepare slide presentations for when I want to show a page but may or may not have access to the Internet during the lecture. Along with simple capture you can edit the page, leave notes, highlights, etc. to really make the page useful. Use this in conjunction with Adblock Plus or you will be capturing a lot of advertisements!</p>
<p>##XMarks</p>
<p>If you use more than one computer with Firefox then you will want some way to synchronize your bookmarks and tabs between machines. <a href="http://www.xmarks.com">XMarks</a> is one of the oldest ways and still one of the best. It handles making sure I can get to my bookmarks no matter what machine I am on and opens Firefox with the last open tabs no matter what machine I am opening it from.</p>
<p>##Zotero</p>
<p>If you are involved in academia at all, preparing papers for teaching or writing papers for school, <a href="http://zotero.org">Zotero</a> is a great tool. With Zotero you can save bookmarks to research sources, save whole PDFs, annotate your sources, add tags and reference data. When you are ready to write the paper it will even output your bibliography entries for you. Where was this thing when I was in school? (probably not conceived of yet as Arpanet still hadn’t gotten off the ground!)</p>
<p>##Pentadactyl</p>
<p>I have written before about my love of the text editor Vim. <a href="http://5digits.org/pentadactyl">Pentadactyl</a> is an implementation of many of the keybindings for Vim to drive Firefox. Want to move to a different tab (i.e. “buffer”)? Just hit “b”, get a list of tabs with numbers and pick the number you want. What to open a new tab? Go to command mode and hit “to”: <code class="highlighter-rouge">:to</code>. Amazing, now my fingers can stay on the home keys more of the time making getting things done faster. One thing to note is that Pentadactyl is under constant development, often Firefox will come out with an update that will break a Pentadactyl install. Usually all you have to do is visit their website and get the latest nightly update to fix any problems.</p>
<p>#Conclusion</p>
<p>So, what are your favorite Firefox plugins? Leave a note in the comments with thoughts, I love trying out new stuff!</p>
Finding a median of a list2014-01-08T00:00:00+00:00http://forbajato.github.io/python/2014/01/08/finding-a-median-of-a-list
<p>#The Puzzle</p>
<p>The description on the website of a median is not too clear to me, fortunately I have a little bit of statistics in my educational background. The median is the middle value of a data set. When you are at the median 50% of the data should be higher than that value, 50% should be lower. In order to find a median you need to have an ordered data set and then simply find the middle value. The rub comes with data sets with an even number of elements. What is the middle value of the list <code class="highlighter-rouge">[1,2,3,4]</code>? Simply take the two values in the center and get their average.</p>
<p>#Odd numbered lists are easy</p>
<p>Given a data set like <code class="highlighter-rouge">[1,2,3,4,5]</code> you can eyeball the solution, the 3 is obviously in the center (with two digits above and 2 digits below). Python can simply check the length of the list:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>l = [1,2,3,4,5]
len(l)
</code></pre>
</div>
<p>In this case Python will return 5. Remember that Python lists are 0 indexed (i.e. the first element is element number 0) so the median in this list will be element number 2. How is 2 related to five? Well it is the whole number portion of the result of dividing 5 in half. In Python 2.7 this is the result of the division <code class="highlighter-rouge">5/2</code> (called “integer division”). If you want the remainder portion you have to do a modulo <code class="highlighter-rouge">5%2</code> (answer will be 1 because 5 divided by 2 is 2 with one left over).</p>
<p>So to get the median all you need is the number in the number 2 position of the list. Even better, all you need is the element in the number 5/2 position of the list.</p>
<p>But wait, where did the 5 come from? Actually, all we need is the element in the number len(l)/2 position of the list. It is time to make a function that will take a list as an input and give us the middle element:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def median(input_list):
return input_list[len(input_list)/2]
</code></pre>
</div>
<p>Simple (at least for the easiest case!). Now, this will only work if a list has an odd number of elements. How can we check? The modulo operator helps here. If you divide an even number by 2 you will never get a remainder. In Python that means the modulo of the division will be 0. We can use that to ensure we are working with an odd numbered list.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def median(input_list):
if len(input_list) % 2 == 1:
return input_list[len(input_list)/2]
</code></pre>
</div>
<p>OK, now we are checking to be sure the list has an odd number of elements in it - the modulo result of an odd number and 2 will be 1 (there should always be a remainder if dividing an odd number by 2, right?). But what if someone gives us a list of digits that are out of order and wants the median? Fortunately Python can sort a list for you. This sort takes place in place, in other words the original list is replaced by a sorted version:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>l = [1,2,3,3,2,1,4,4,3,2]
l.sort()
</code></pre>
</div>
<p>This results in the list l now being <code class="highlighter-rouge">[1,2,2,2,3,3,3,4,4]</code>. Once the list is sorted it is easy to pass to our growing function to get the median:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def median(input_list):
input_list.sort()
if len(input_list) % 2 == 1:
return input_list[len(input_list)/2]
</code></pre>
</div>
<p>Almost there, one possible case to consider first - lists with an even number of elements.</p>
<p>#What about even numbered lists?</p>
<p>What if the list we need to investigate has an even number of elements? First of all, how can we find out? Simple - just check the modulo. If the modulo result of dividing the length of the list by 2 is 0 (i.e. there is no remainder) you know the list has an even number of elements.</p>
<p>The median of a list with an even number of elements is simply the average of the two middle most elements. How can you use list methods to find the two middle most elements? Consider the simple list <code class="highlighter-rouge">[1,2,3,4]</code>. This list has a len of 4. If you divide the len by 2 you get 2. If you chose the element number 2 (from len(list)/2) you would get 3. So you are half way there. All you need is the element before that one and then take the average. We can modify our function one more time (if statements just cry out for an <code class="highlighter-rouge">else</code> clause don’t they?):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def median(input_list):
input_list.sort()
if len(input_list) % 2 == 1:
return input_list[len(input_list)/2]
else:
return (input_list[len(input_list)/2]+input_list[len(input_list)/2 -1])/2.0
</code></pre>
</div>
<p>You will note we used the 2.0 in the final statement, this is to ensure that Python uses real division, giving us a number with a decimal as the result.</p>
<p>There are ways to make this shorter but, to me, the code then gets more difficult to read. One of the advantages of a clear language like Python is that you can tell what is happening just by reading the code, thus don’t have to document quite as much.</p>
<p>Keep hacking!</p>
i3 Challenge2014-01-07T00:00:00+00:00http://forbajato.github.io/blog/2014/01/07/i3-challenge
<p>#Rationale</p>
<p>Tiling window managers are interesting. I have tried out dwm, awesome and i3 in the past but only ever really spent time with awesome. Awesome has had a tendency to make rather jarring changes in how they configure the WM and I have never really mastered it. When I recently checked out <a href="http://i3wm.org">i3</a> I decided to give it a go.</p>
<p>Why use a tiling window manager? Well, I find that I spend most of my time with my applications maximized. Even though KDE has a great desktop with widgets galore available, I find that I don’t use them much because my browser or terminal is maximized and covers up all those cool widgets. Since I am covering up those widgets anyway, why use them?</p>
<p>I love Vim, I love that I can use the editor without my fingers leaving the keyboard. I have installed Pentadactyl to allow a similar functionality in Firefox. What if the whole window manager could be driven with keyboard only rather than having to reach for the mouse every couple of seconds?</p>
<p>Add to that my tendency to use computers way past their “sell by” dates (my eeePC is over 7 years old). Tiling window managers are well known for being light on resources. All in all I have good reason to want to check out this sort of interface.</p>
<p>#Installing i3</p>
<p>In Ubuntu installing i3 was a breeze:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo apt-get install i3
</code></pre>
</div>
<p>There is a metapackage that handles installing i3 and some other goodies (like i3lock, i3status and dmenu). Dmenu is essential as far as I am concerned because otherwise you have to launch every application through a terminal, either spawning a lot of terminals or using the “&” switch a lot at the end of your commands.</p>
<p>#Configuration</p>
<p>i3 comes with the default $mod key being either the <code class="highlighter-rouge"><Alt></code> key or the <code class="highlighter-rouge"><Super></code> key (the one with an Ubunutu logo on my System76 laptop but probably has some Windows related logo on other keyboards). There is a great overview of configuration and use of i3 on Youtube posted by <a href="http://www.youtube.com/watch?v=yAq_Enj_d2Q">gotbletu</a>. I started there and have made some tweaks that fit my work flow.</p>
<p>I like using the <code class="highlighter-rouge"><Alt></code> key for my $mod key so at the top of the .i3/config file you set:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>set $mod Mod1
</code></pre>
</div>
<p>If you want to use the <code class="highlighter-rouge"><Super></code> key instead just set that to Mod4.</p>
<p>I love Synapse, the program launcher that I have used for years in XFCE4, KDE and even Unity. Firing it up is now part of my muscle memory and loosing that would be a shame. So to start dmenu I need to have it use the same key combination that I use to access Synapse in the other desk top environments:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>bindsym Mod4+space exec dmenu_run
</code></pre>
</div>
<p>I complete agree with gotbletu, making the moving focus to be vim compatible is very helpful. Search down the config file for the line “# change focus” and make some changes:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>bindsym $mod+h focus left
bindsym $mod+j focus down
bindsym $mod+k focus up
bindsym $mod+l focus right
</code></pre>
</div>
<p>I made the similar changes in the “# move focused window” section.</p>
<p>The maker of the youtube video referenced above sees things differently than I do. I think that when you do a “split vertical” you should end up with two windows, one stacked on top of the other in vertical alignment. Apparently the makers of i3 <a href="http://i3wm.org/docs/userguide.html#_tree" title="The Tree in i3">agree</a> so they set up the splits with $mod+v giving a vertical split and $mod+h giving a horizontal split. That would be great but we just defined $mod+h to focus left. If you tried to restart i3 with the new config after making the focus changes you got an error, i3 won’t let you have two actions bound to the same keybindings (thanks to the devs for that!). So I remapped the split horizontal to $mod+b, the ‘b’ key is right next to the ‘v’ key in a qwerty keyboard layout so that made sense to me.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>bindsym $mod+b split h
</code></pre>
</div>
<p>Now I love keyboard bindings and try to learn a few bindings for common tasks on my most used applications on a routine basis. My hands have gotten used to the <code class="highlighter-rouge"><Alt>fd</code> combination to send unsent mail in Thunderbird. Unfortunately i3 has $mod+f mapped to go full screen. Easy enough to change, we will just map that to the Ubuntu key instead of the <code class="highlighter-rouge"><Alt></code> key:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>bindsym Mod4+f fullscreen
</code></pre>
</div>
<p>Notice to map it to the <code class="highlighter-rouge"><Super></code> key I used the full name of the key, not the variable (i.e. not “$mod4”).</p>
<p>I have been using a 4 workspace model with KDE for sometime now - terminal (either in a workspace or using guake), web (browser, email client, etc.), fun (gotta have a place for Steam!) and media. With i3 you can name those workspaces and the key combinations to jump to them. Search down the config file for the “# switch to workspace” line and add:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>bindsym $mod+1 workspace 1: terminal
bindsym $mod+2 workspace 2: web
bindsym $mod+3 workspace 3: fun
bindsym $mod+4 workspace 4: media
</code></pre>
</div>
<p>It is important to note that the name of the workspace here starts with the digit following the word “workspace”, anything you put out there will name the space. Another thing to keep in mind is that you are creating new workspaces with this set of commands. What that means is that even though you have a workspace named “1: terminal” that doesn’t mean you can’t have another workspace named “1”. I discovered this when trying to move an application to workspace 1 when I already had workspace “1: terminal” - i3 created another workspace named “1” and moved my application there. As such, in order to avoid confusion you may want to make similar changes to the move to workspace section.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>bindsym $mod+Shift+1 move container to workspace 1: terminal
bindsym $mod+Shift+2 move container to workspace 2: web
bindsym $mod+Shift+3 move container to workspace 3: fun
bindsym $mod+Shift+4 move container to workspace 4: media
</code></pre>
</div>
<p>It sure would be nice if you could define certain applications to always open in certain workspaces, wouldn’t it? i3 (and other tiling window managers as well) has that covered.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>assign [class="Mcomix"] 4: media
assign [class="Calibre"] 4: media
assign [class="Gpodder"] 4: media
assign [class="Vlc"] 4: media
assign [class="Steam"] 3: fun
assign [class="Firefox"] 2: web
assign [class="Thunderbird"] 2: web
</code></pre>
</div>
<p>How do you find what to put in the class=”?” section? Use xprop or xwininfo. All you have to do is fire up xprop in a terminal then click on the window you need the class information for. Look through the output for the line that starts WM_CLASS(STRING), it will look something like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>WM_CLASS(STRING) = "Navigator", "Firefox"
</code></pre>
</div>
<p>Use one of those names in the class variable above.</p>
<p>Now I like to cycle though workspaces sometimes rather than jumping directly to the space I need. i3 doesn’t do that by default but let’s face it - we control the machine, not the other way around! So the i3 devs gave a way to set this up:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>bindsym $mod+n workspace next
bindsym $mod+p workspace previous
</code></pre>
</div>
<p>Ain’t open source wonderful?!?</p>
<p>Now, with a tiling window manager you won’t be seeing your wallpaper often but just in case you want some interesting wallpaper you can use feh to handle that for you:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>exec feh --bg-scale /path/to/wallpaper/file.png
</code></pre>
</div>
<p>There are a number of applications that I can’t live without, you can set them to autostart when i3 fires up:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> # start clipboard manager
exec parcellite
# gotta turn off that mousepad when I am typing, glad KDE came with synaptiks to make that easy to configure!
exec synaptiks
# there are probably people who can live without a drop down terminal, I am not one of them
exec sleep 3 && guake
</code></pre>
</div>
<p>There are still a few things I haven’t figured out yet, chief among them is volume control with the multimedia keys. Other than that little annoyance I am thoroughly enjoying my i3 experience (having used it exclusively for a week), even my wife is tolerating it! If you have other thoughts on tweaks for i3 or config ideas please feel free to leave them in the comments.</p>
Favorite Vim Plugins2013-12-31T00:00:00+00:00http://forbajato.github.io/blog/2013/12/31/favorite-vim-plugins
<p>#Introduction</p>
<p>I love Vim. You don’t have to go too far on the internet to see that there are two raging camps out there. I admit to never having tried emacs but, having spent so much time learning and tweaking vim I am unlikely to try it now. What I like about vim:</p>
<ol>
<li>It gets out of my way - OK, there is a learning curve. Once you get basic movement, search, copy and replace down though vim stays out of your way. The screen is blank with a flashing cursor - nothing to distract.</li>
<li>It is amazingly extensible - I suspect this is true for emacs as well. I find that whenever I am doing a task repeatedly there is either an easy way to automate it in vim or someone has already written a plugin to make it faster.</li>
<li>Vi is everywhere - most all Linux based systems will have some form of Vi installed by default. Vim builds on Vi but knowing the Vim basics means I can administer systems that have just started up easily and using familiar keybindings.</li>
</ol>
<p>As great as Vim is there are things it doesn’t do easily, that is where plugins come in. I have never written a plugin myself but since there are so many smart people out there who make their plugins available I haven’t needed to write anything from scratch. Following is a list of my most used plugins.</p>
<p>#Pathogen - where it all should begin</p>
<p>Getting Vim plugins to work is not generally difficult but sometimes there are a number of files that have to be put in the correct spots in the file tree. Along came <a href="https://github.com/tpope/vim-pathogen">Pathogen</a>. This amazing little plugin allows you to install and manage plugins for Vim in their own private directories. After a bit of configuring (see the Pathogen git repo for instructions) installing a plugin (like, say NerdTree) is simple:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cd ~/.vim/bundle
git clone https://github.com/scrooloose/nerdrtee.git
</code></pre>
</div>
<p>Simple!</p>
<p>#Plugins I installed using Pathogen</p>
<p>Before I discovered Pathogen I installed a few plugins and will discuss those below but first let’s deal with the Pathogen managed plugins.</p>
<p>##Multiple Search 2</p>
<p>Ever wish you could do a search for a term then search a different term without loosing the highlighting on the first term? If not, move on, nothing to see here - but if so, checkout <a href="https://github.com/vim-scripts/MultipleSearch2.vim">MultipleSearch2</a>. Really handy if you need to do multiple searches through your document and keep them both accessible.</p>
<p>##Nerd Commenter</p>
<p>Sometimes when you are coding you need to comment out something for the purposes of debugging. I know when I run tests I just need to get some pesky code out of the way to see where things are going wrong.</p>
<p>Enter <a href="https://github.com/scrooloose/nerdcommenter">NerdCommenter</a> - highlight some text you want commented, hit a quick key combination and you have commented out your code. This thing even knows how to comment out different types of code! I use this every coding session.</p>
<p>##Nerd Tree</p>
<p>OK, you are editing some code, building your killer Django app and you realize you need to check the model (does that ever happen to anyone else?). If the directory tree is simple you can just open the file you need but if it is a complicated directory tree using a tree navigator can be really helpful. <a href="https://github.com/scrooloose/nerdtree">NerdTree</a> fits the bill.</p>
<p>##Tagbar</p>
<p>Sometimes I need to move through my code quickly to find a class, function, variable, etc. Folds can help with this but wouldn’t it be nice if something could keep track for you automatically? <a href="https://github.com/majutsushi/tagbar">Tagbar</a> opens a window (on the right by default) with a list of the code tags in your file, move up and down the window to jump directly to the class, function, etc. you need to edit.</p>
<p>##Snipmate</p>
<p>While I haven’t used this in coding much (I do a lot of python coding and don’t find a need for this there) I do use it in writing papers, developing slides and writing tests in LaTeX. <a href="https://github.com/msanders/snipmate.vim">SnipMate</a> has a host of predefined code snipits that it can insert into your text with a simple command. When you are writing slide environments for beamer or doing test questions for eqexam (LaTeX plugins) this can save lots of time, also helps to ensure I get the tags correct when I have not used the language for a while.</p>
<p>##Vim-Surround</p>
<p>Like snipmate this one is designed to get you coding quickly. Anyone who does html will benefit from this - <a href="https://github.com/tpope/vim-surround">Vim-Surround</a> will simple surround your highlighted text with a particular tag.</p>
<p>#Essential non-pathogen plugins</p>
<p>I came late to the pathogen party so I have a few plugins I still use from before. It may be that these are usable as pathogen plugins now but I installed the manually.</p>
<p>##<a href="http://www.vim.org/scripts/script.php?script_id=790">Python.vim</a></p>
<p>Most programming languages have their own plugins to help with syntax highlighting, code testing, etc. Python is no different.</p>
<p>##<a href="https://github.com/vim-scripts/potwiki.vim">Potwiki</a></p>
<p>Ever wanted to turn Vim into a text based wiki? This one will handle it for you!</p>
<p>##<a href="https://github.com/vim-scripts/TaskList.vim">Tasklist</a></p>
<p>Add GTD like task list support to your text based wiki and you have a powerful productivity tool (remember - greater productivity means more time for Steam games).</p>
<p>There you have it - my favorite and most used Vim plugins. There are tons more, if I missed one that you particularly love make your case for it in the comments!</p>
Finding Non Unique Elements in a list with python2013-12-30T00:00:00+00:00http://forbajato.github.io/python/2013/12/30/finding-non-unique-elements-in-a-list-with-python
<p>#Introduction</p>
<p>I am always looking for ways to keep learning different things. I love programming and my most common language of choice is python (though I have played with arduino, processing, javascript and bash). Fortunately there is no shortage of web based tutorials and challenges to get you going in Python programming.</p>
<p>Recently I stumbled on the <a href="http://www.checkio.org">Checkio</a> site. This is like the <a href="http://pythonchallenge.com">Python Challenge</a> site in that there are staged puzzles that you must solve in order to move forward. Unlike the Python Challenge each level has a number of puzzles that you can choose from so if you are flailing away at one problem you can always switch to another to keep moving forward and keep learning.</p>
<p>#The puzzle</p>
<p>The first puzzle on the site seems to be the “Non-unique Elements” puzzle. In it you have to take a list of arbitrary length and find all the elements in it that are not unique. For instance the list <code class="highlighter-rouge">[1,1,3,3,2]</code> would return a list of only the non-unique elements <code class="highlighter-rouge">[1,1,3,3]</code> and discard the 2.</p>
<p>#First solution (i.e. “The inelegant one!”)</p>
<p>I always fall into this trap - thinking I have to build a list from another list. It works but is clunky and makes the code rather long. Anyway this worked:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def checkio(data):
ans = []
for r in data:
if data.count(r) > 1:
ans.append(r)
return ans
</code></pre>
</div>
<p>All the function is doing is setting up an empty list (to collect the answers) then looping through the list passed in to it and counting the number of times each element shows up in the original list. If that count is greater than 1 (i.e. the element is non-unique) then the element is appended to the answer list. Finally the answer list is returned.</p>
<p>This has a number of problems.</p>
<ol>
<li>It is rather long - too many lines of code for something that could be done in a much simpler way.</li>
<li>Nagging feeling that there is a better way - ever have that nagging feeling that you are doing something you have done wrong before and know there is a better way? This gave me that feeling too.</li>
<li>Needlessly counting every element as it is encountered - why should you have to count the first element again when it is encountered the second time in the list? You already know it is in the answer list. For short lists this is not important but if you are dealing with large datasets this could be annoying and time consuming.</li>
</ol>
<p>#Using a list comprehension</p>
<p>OK, the nagging went away when I realized the better solution is with a list comprehension. You think of a list comprehension as a way to make a list from a data set and should always think of it when you have the pattern:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>ans=[]
<do something>
ans.append(list_element)
</code></pre>
</div>
<p>If you are looping and doing a lot of “loop.appending” there is a better way. List comprehensions can also have a conditional tied to them so here is a much better way (still not the best, you have to solve the puzzle yourself to see the best submitted answer!).</p>
<div class="highlighter-rouge"><pre class="highlight"><code>def checkio_list_comprehension(data):
ans = [r for r in data if data.count(r) > 1]
return ans
</code></pre>
</div>
<p>So much simpler, easier to read (assuming you know about list comprehensions which, now you do!) and easier to debug. You could even set the return to the list comprehension and eliminate a line of code further.</p>
<p>I still don’t have a great answer to how to keep from checking repeated elements repeatedly. If you have a thought leave a comment!</p>
Setting up a github blog using Jekyll Bootstrap and Disqus2013-12-18T00:00:00+00:00http://forbajato.github.io/blog/2013/12/18/setting-up-a-github-blog-using-jekyll-bootstrap-and-disqus
<p>#Rationale</p>
<p>Ever have a problem with your computer, robot or other tech toy? If you are like me, you go to <a href="http://duckduckgo.com">Duck Duck Go</a> and search out an answer, try out a few of the first hits, find the perfect solution, tutorial, etc., fix the problem then promptly forget it. Next time you run into the problem you have to cycle through the steps again. I am tired of the “lather, rinse, repeat” cycle of technology problem solving - time to start documenting all the best answers someplace that I know I can find again, in a system that I can control and one that I can use to get feedback from a community of nerds to improve my process. Thus this blog was born!</p>
<p>I had heard about Twitter bootstrap before and what it promised sounded really good - a way to set up the design of websites so that they look good (something important to me but not important enough to inspire me to figure out the ins and outs of design) and can be rendered on multiple platforms (computer, tablet, phone) nicely. When I found that Github has a thing that allows static pages websites and that you can use Jekyll-Bootstrap to make the sites I was sold. I had to figure out how to set this up and use it to start a tech blog to document all my tutorial finds, technology configuration wins, etc.</p>
<p>This first post on the new blog will document how to set up the beast. In the process of starting this I tried multiple tutorials on Jekyll-Bootstrap (and variations) as well as Pelican. I liked the idea of Pelican since it is Python based and I know a little Python but in the end Jekyll-Bootstrap turned out to be the one that I managed to make work. Getting this hosted on Github was a natural fit.</p>
<p>Without further adieu, here is how I managed to make it work. As always, your mileage may vary but if you have questions on the process documented here or ideas to improve the process please leave a comment!</p>
<p>#Prepare Github</p>
<p><a href="http://github.com">Github</a> has a static web pages hosting service. Through this service you can host static web sites, that would be sites that do not interact a lot with the user but simply display information. If you want a simple blog this is a great way to go. Add to that you can control your own content, develop your content locally and manage versions (through git) and you have a pretty great package.</p>
<p>If you don’t have a github account yet you will want to go <a href="http://github.com">here</a> and sign up for one. The accounts are free and afterwards you can put all kinds of projects under version control. Do keep in mind that if you want an account that hides code (or blog articles) you will need a paid account. If you want private repositories for code development you may want to try <a href="http://bitbucket.org">Bitbucket</a>. Bitbucket allows git and mercurial repos but doesn’t have the static webpage hosting service.</p>
<p>Once you have your github account you will need a repository to host the blog. Click the “Create a new repo” icon (top right hand side, next to your username). Name your repo something like USERNAME.github.com. It turns out you can probably put most anything you want in for USERNAME but most examples use your github username, I know that works as that is what I did.</p>
<p>At this point the repo isn’t fully initialized - there are no files under version control (unless you chose to initialize it when creating by checking the “Initialize this repository with a README” box when creating the repo). Doing it this way is probably the easiest as you don’t end up with colliding repos. Next you want to go to the Jekyll-Bootstrap startup page to get things going with Jekyll-Bootstrap.</p>
<p>If none of this is working and you need a little help getting started or getting familiar with git, github has some great <a href="https://help.github.com">help pages</a>. There are also some great tutorials for using git around the web, I started with the free one from <a href="http://codeschool.com/courses/try-git">Code School</a>.</p>
<p>#Set up Jekyll-Bootstrap</p>
<p>Once you have your github repo set up it is time to head over to <a href="http://jekyllbootstrap.com">Jekyll Bootstrap</a>. Jekyll Bootstrap is designed to get you up and running with a blog on github as fast as possible. I have tried ruhoh (linked at the bottom of the JB front page) but that didn’t really work well for me. Jekyll Bootstrap had me up and running, theming and creating content very quickly. My needs are simple, again, your mileage may vary.</p>
<p>Click the “0 to Blog in 3 Minutes” button on the Jekyll Bootstrap page. This will send you to a form that will start giving you code to enter into the command line to get JB set up. If you put your github username in the box at the top of the page it will even customize the code for you.</p>
<p>You will notice that step two involves some git commands - again, if you are not familiar with git you may want to do the tutorial linked above to get your head around git. If you follow these steps exactly and have already initialized the repo with a README you may run into a problem when you try to push the local repo to github.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>git push origin master
</code></pre>
</div>
<p>You will get an error stating that the repo on github’s server is ahead of the repo on your local machine. You will need to do a pull, merge, then push:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>git pull
git merge
git push origin master
</code></pre>
</div>
<p>That should set things straight again.</p>
<p>I didn’t bother with the “Run Jekyll locally” steps but if you want to (and are using Ubuntu) you will need to install a few things first:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo apt-get install ruby rubygems
</code></pre>
</div>
<p>I don’t know enough ruby to say but after installing these it looks like installing other ruby packages is simply</p>
<div class="highlighter-rouge"><pre class="highlight"><code>gem install <package>
</code></pre>
</div>
<p>I found installing rake (kind of like make but for ruby packages as best I can tell) using the Ubuntu repos was helpful.</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo apt-get install rake
</code></pre>
</div>
<p>#Creating posts</p>
<p>Obviously you made the blog to create posts. Starting a post is really easy:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>rake post title="Hello World"
</code></pre>
</div>
<p>This will create a markdown file in <code class="highlighter-rouge">_posts</code> directory with the date and post name followed by the markdown extension. An example of what you will have:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>---
layout: post
title: "Hello World"
description: ""
category: ""
tags: []
---
</code></pre>
</div>
<p>Now you will just need to edit that text file to finish the post. Put in a short description of the post under “description:”, add in a category and some tags to track it. Finally the post will need some text. Here is where you get to put your Markdown prowess to work.</p>
<p>##So you say you don’t know Markdown?</p>
<p>Say it ain’t so! Markdown is a fabulous text markup language that allows you to write your text in human readable form and then translate it into multiple formats. Someday I will do a post extolling the virtue of <a href="http://johnmacfarlane.net/pandoc/">Pandoc</a> for making web pages, beamer slide shows and pdf documents out of markdown files. For now though you don’t need to worry about any of that - just need to learn enough Markdown to make a renderable post.</p>
<p>Markdown text is all in plain text (can you say “easy grepping”? I knew you could!) with a few symbols along the way to help you. Precede your line of text with a hash mark (#) and you get a headline, increase the number of hash marks for decreasing levels of headlines:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>#Heading 1
##Heading 2
###Heading 3
</code></pre>
</div>
<p>Want to make a list? Do you need numbers with that?</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1. Item one
1. Item two
1. Item three
</code></pre>
</div>
<p>Don’t worry - the server will fix the numbering (getting an idea of how easy this is to use?). If you don’t need enumerated lists you just:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>* Red
* Blue
* Yellow
</code></pre>
</div>
<p>There is much more to Markdown than this, head over to <a href="http://daringfireball.net/projects/markdown/basics">Daring Fireball</a> for more. The beautiful thing about using Jekyll is that when you push your markdown files (in the <code class="highlighter-rouge">_posts</code> directory and with a <code class="highlighter-rouge">.md</code> or <code class="highlighter-rouge">.markdown</code> file extension) to github, their server will render the files into blog posts for you - magic!</p>
<p>#Set up comments with Disqus</p>
<p>What is a blog without comments? On a blog like this one I need comments because I am talking about how I do things - how else can people get further help or make suggestions to improve my processes without a form of comments?</p>
<p>Jekyll allows for a number of possible comment engines. Looking in the <code class="highlighter-rouge">_config.yml</code> file you see a section that looks like this:</p>
<div class="highlighter-rouge"><pre class="highlight"><code># Settings for comments helper
# Set 'provider' to the comment provider you want to use.
# Set 'provider' to false to turn commenting off globally.
#
comments :
provider : disqus
disqus :
short_name : yourdisqusshortname
livefyre :
site_id : 123
intensedebate :
account : 123abc
facebook :
appid : 123
num_posts: 5
width: 580
colorscheme: light
</code></pre>
</div>
<p>This blog uses <a href="http://disqus.com">Disqus</a> so to set up comments I went to their site and set up an account. In setting things up I had to give the blog a name, give Disqus information about the blog (minimal) and set a short name. When I had that I could put the shortname in the <code class="highlighter-rouge">_config.yml</code> file. Next you will need to follow the link for setup instructions for “Universal Code”. This will give you a segment of html that you can place in your website pages to connect your Disqus comment account with the blog. Put this block of code in your <code class="highlighter-rouge">_layouts/posts.html</code> file. This file is a template that grabs code from various places, I simply copied the html from Disqus and pasted it at the bottom of the posts.html file, this places the Disqus comment box at the bottom of all post pages the site generates.</p>
<p>#Checking the post to be sure it will render</p>
<p>Before you push this bad boy to the github servers you may want to check and be sure it is compiling correctly. If you push a markdown page up to github and it isn’t compiling correctly github simply won’t make changes to your website and you will be stuck wondering what is going on (like I have been for the past hour or so).</p>
<p>Checking this requires that you have Jekyll installed locally. I decided to use the gems (I think of these like pip install commands in Python and assume you are getting the freshest packages that way).</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo gem install jekyll
</code></pre>
</div>
<p>This install will take a few minutes as a few dependencies need to come along with the jekyll package. When it is installed you can start a local jekyll instance with a server to see if your markdown compiles correctly. You need to start this server from the directory that has the site in it:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cd /home/USERNAME/git-repos/USERNAME.github.com
jekyll serve
</code></pre>
</div>
<p>If you get a lot of error messages you may want to parse through this output with a trace:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>jekyll serve -trace | less
</code></pre>
</div>
<p>I made a few errors and each error takes up several lines so I piped the output through less so I could make changes.</p>
<p>Once you have the markdown on your post correctly written you should see this when you start the server:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Configuration file: /home/USERNAME/git-repos/USERNAME.github.com/_config.yml
Source: /home/USERNAME/git-repos/USERNAME.github.com
Destination: /home/USERNAME/git-repos/USERNAME.github.com/_site
Generating... Maruku#to_s is deprecated and will be removed or changed in a near-future version of Maruku.
done.
Server address: http://0.0.0.0:4000
Server running... press ctrl-c to stop.
</code></pre>
</div>
<p>Errors I had made:</p>
<ol>
<li>malformed URL (hope that will improve as I work with markdown more)</li>
<li>not putting a <code class="highlighter-rouge">_config.yml</code> in backticks (the preceding underscore is a special character in markdown and needs to be escaped with backticks)</li>
</ol>
<p>Now you are ready to update the local repo, commit and push to github.</p>
<p>#Getting the posts on the blog</p>
<p>One of the great things about this approach is you can use your tools on your computer and then push the content to github for hosting. When you have your posts ready all you need to do is be sure the files are staged for commit:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>git status
</code></pre>
</div>
<p>This will show you if there are files that are untracked in your local directory. Add them with</p>
<p>git add .</p>
<p>Then commit and send to github:</p>
<p>git commit -am “Commit message, make it something descriptive or you will be sorry later!”
git push origin master</p>
<p>Note: the -a switch on <code class="highlighter-rouge">git commit</code> makes sure that all files that are tracked but have changed will also be committed, one of the annoyances of git (for a mercurial user) is that changed files have to be added this way in order to get them to still be tracked.</p>
<p>#Conclusion</p>
<p>There is much more to Jekyll Bootstrap (themes, templates, Liquid, etc.), Markdown (tons of options to format the text, use Pandoc to make lots of different outputs, etc.), and Disqus (connect more blog and other sites for comments, control users, approve comments, etc.). This should have you up and running with a blog pretty quickly though. Now that you have something - go create!</p>
My Hello World2013-12-17T00:00:00+00:00http://forbajato.github.io/lessons/2013/12/17/my-hello-world
<p>This is an attempt to write a blog post in markdown that can then deploy to my github static blog. If this thing appears then I know it worked. I don’t know if markdown can handle single quotes properly. Let’s test them.</p>
Jekyll Introduction2011-12-29T00:00:00+00:00http://forbajato.github.io/lessons/2011/12/29/jekyll-introduction
<p>This Jekyll introduction will outline specifically what Jekyll is and why you would want to use it.
Directly following the intro we’ll learn exactly <em>how</em> Jekyll does what it does.</p>
<h2 id="overview">Overview</h2>
<h3 id="what-is-jekyll">What is Jekyll?</h3>
<p>Jekyll is a parsing engine bundled as a ruby gem used to build static websites from
dynamic components such as templates, partials, liquid code, markdown, etc. Jekyll is known as “a simple, blog aware, static site generator”.</p>
<h3 id="examples">Examples</h3>
<p>This website is created with Jekyll. <a href="https://github.com/mojombo/jekyll/wiki/Sites">Other Jekyll websites</a>.</p>
<h3 id="what-does-jekyll-do">What does Jekyll Do?</h3>
<p>Jekyll is a ruby gem you install on your local system.
Once there you can call <code class="highlighter-rouge">jekyll --server</code> on a directory and provided that directory
is setup in a way jekyll expects, it will do magic stuff like parse markdown/textile files,
compute categories, tags, permalinks, and construct your pages from layout templates and partials.</p>
<p>Once parsed, Jekyll stores the result in a self-contained static <code class="highlighter-rouge">_site</code> folder.
The intention here is that you can serve all contents in this folder statically from a plain static web-server.</p>
<p>You can think of Jekyll as a normalish dynamic blog but rather than parsing content, templates, and tags
on each request, Jekyll does this once <em>beforehand</em> and caches the <em>entire website</em> in a folder for serving statically.</p>
<h3 id="jekyll-is-not-blogging-software">Jekyll is Not Blogging Software</h3>
<p><strong>Jekyll is a parsing engine.</strong></p>
<p>Jekyll does not come with any content nor does it have any templates or design elements.
This is a common source of confusion when getting started.
Jekyll does not come with anything you actually use or see on your website - you have to make it.</p>
<h3 id="why-should-i-care">Why Should I Care?</h3>
<p>Jekyll is very minimalistic and very efficient.
The most important thing to realize about Jekyll is that it creates a static representation of your website requiring only a static web-server.
Traditional dynamic blogs like Wordpress require a database and server-side code.
Heavily trafficked dynamic blogs must employ a caching layer that ultimately performs the same job Jekyll sets out to do; serve static content.</p>
<p>Therefore if you like to keep things simple and you prefer the command-line over an admin panel UI then give Jekyll a try.</p>
<p><strong>Developers like Jekyll because we can write content like we write code:</strong></p>
<ul>
<li>Ability to write content in markdown or textile in your favorite text-editor.</li>
<li>Ability to write and preview your content via localhost.</li>
<li>No internet connection required.</li>
<li>Ability to publish via git.</li>
<li>Ability to host your blog on a static web-server.</li>
<li>Ability to host freely on GitHub Pages.</li>
<li>No database required.</li>
</ul>
<h1 id="how-jekyll-works">How Jekyll Works</h1>
<p>The following is a complete but concise outline of exactly how Jekyll works.</p>
<p>Be aware that core concepts are introduced in rapid succession without code examples.
This information is not intended to specifically teach you how to do anything, rather it
is intended to give you the <em>full picture</em> relative to what is going on in Jekyll-world.</p>
<p>Learning these core concepts should help you avoid common frustrations and ultimately
help you better understand the code examples contained throughout Jekyll-Bootstrap.</p>
<h2 id="initial-setup">Initial Setup</h2>
<p>After <a href="/index.html#start-now">installing jekyll</a> you’ll need to format your website directory in a way jekyll expects.
Jekyll-bootstrap conveniently provides the base directory format.</p>
<h3 id="the-jekyll-application-base-format">The Jekyll Application Base Format</h3>
<p>Jekyll expects your website directory to be laid out like so:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>.
|-- _config.yml
|-- _includes
|-- _layouts
| |-- default.html
| |-- post.html
|-- _posts
| |-- 2011-10-25-open-source-is-good.markdown
| |-- 2011-04-26-hello-world.markdown
|-- _site
|-- index.html
|-- assets
|-- css
|-- style.css
|-- javascripts
</code></pre>
</div>
<ul>
<li>
<p><strong>_config.yml</strong>
Stores configuration data.</p>
</li>
<li>
<p><strong>_includes</strong>
This folder is for partial views.</p>
</li>
<li>
<p><strong>_layouts</strong>
This folder is for the main templates your content will be inserted into.
You can have different layouts for different pages or page sections.</p>
</li>
<li>
<p><strong>_posts</strong>
This folder contains your dynamic content/posts.
the naming format is required to be <code class="highlighter-rouge">@YEAR-MONTH-DATE-title.MARKUP@</code>.</p>
</li>
<li>
<p><strong>_site</strong>
This is where the generated site will be placed once Jekyll is done transforming it.</p>
</li>
<li>
<p><strong>assets</strong>
This folder is not part of the standard jekyll structure.
The assets folder represents <em>any generic</em> folder you happen to create in your root directory.
Directories and files not properly formatted for jekyll will be left untouched for you to serve normally.</p>
</li>
</ul>
<p>(read more: <a href="https://github.com/mojombo/jekyll/wiki/Usage">https://github.com/mojombo/jekyll/wiki/Usage</a>)</p>
<h3 id="jekyll-configuration">Jekyll Configuration</h3>
<p>Jekyll supports various configuration options that are fully outlined here:
(<a href="https://github.com/mojombo/jekyll/wiki/Configuration">https://github.com/mojombo/jekyll/wiki/Configuration</a>)</p>
<h2 id="content-in-jekyll">Content in Jekyll</h2>
<p>Content in Jekyll is either a post or a page.
These content “objects” get inserted into one or more templates to build the final output for its respective static-page.</p>
<h3 id="posts-and-pages">Posts and Pages</h3>
<p>Both posts and pages should be written in markdown, textile, or HTML and may also contain Liquid templating syntax.
Both posts and pages can have meta-data assigned on a per-page basis such as title, url path, as well as arbitrary custom meta-data.</p>
<h3 id="working-with-posts">Working With Posts</h3>
<p><strong>Creating a Post</strong>
Posts are created by properly formatting a file and placing it the <code class="highlighter-rouge">_posts</code> folder.</p>
<p><strong>Formatting</strong>
A post must have a valid filename in the form <code class="highlighter-rouge">YEAR-MONTH-DATE-title.MARKUP</code> and be placed in the <code class="highlighter-rouge">_posts</code> directory.
If the data format is invalid Jekyll will not recognize the file as a post. The date and title are automatically parsed from the filename of the post file.
Additionally, each file must have <a href="https://github.com/mojombo/jekyll/wiki/YAML-Front-Matter">YAML Front-Matter</a> prepended to its content.
YAML Front-Matter is a valid YAML syntax specifying meta-data for the given file.</p>
<p><strong>Order</strong>
Ordering is an important part of Jekyll but it is hard to specify a custom ordering strategy.
Only reverse chronological and chronological ordering is supported in Jekyll.</p>
<p>Since the date is hard-coded into the filename format, to change the order, you must change the dates in the filenames.</p>
<p><strong>Tags</strong>
Posts can have tags associated with them as part of their meta-data.
Tags may be placed on posts by providing them in the post’s YAML front matter.
You have access to the post-specific tags in the templates. These tags also get added to the sitewide collection.</p>
<p><strong>Categories</strong>
Posts may be categorized by providing one or more categories in the YAML front matter.
Categories offer more significance over tags in that they can be reflected in the URL path to the given post.
Note categories in Jekyll work in a specific way.
If you define more than one category you are defining a category hierarchy “set”.
Example:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>---
title : Hello World
categories : [lessons, beginner]
---
</code></pre>
</div>
<p>This defines the category hierarchy “lessons/beginner”. Note this is <em>one category</em> node in Jekyll.
You won’t find “lessons” and “beginner” as two separate categories unless you define them elsewhere as singular categories.</p>
<h3 id="working-with-pages">Working With Pages</h3>
<p><strong>Creating a Page</strong>
Pages are created by properly formatting a file and placing it anywhere in the root directory or subdirectories that do <em>not</em> start with an underscore.</p>
<p><strong>Formatting</strong>
In order to register as a Jekyll page the file must contain <a href="https://github.com/mojombo/jekyll/wiki/YAML-Front-Matter">YAML Front-Matter</a>.
Registering a page means 1) that Jekyll will process the page and 2) that the page object will be available in the <code class="highlighter-rouge">site.pages</code> array for inclusion into your templates.</p>
<p><strong>Categories and Tags</strong>
Pages do not compute categories nor tags so defining them will have no effect.</p>
<p><strong>Sub-Directories</strong>
If pages are defined in sub-directories, the path to the page will be reflected in the url.
Example:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>.
|-- people
|-- bob
|-- essay.html
</code></pre>
</div>
<p>This page will be available at <code class="highlighter-rouge">http://yourdomain.com/people/bob/essay.html</code></p>
<p><strong>Recommended Pages</strong></p>
<ul>
<li><strong>index.html</strong>
You will always want to define the root index.html page as this will display on your root URL.</li>
<li><strong>404.html</strong>
Create a root 404.html page and GitHub Pages will serve it as your 404 response.</li>
<li><strong>sitemap.html</strong>
Generating a sitemap is good practice for SEO.</li>
<li><strong>about.html</strong>
A nice about page is easy to do and gives the human perspective to your website.</li>
</ul>
<h2 id="templates-in-jekyll">Templates in Jekyll</h2>
<p>Templates are used to contain a page’s or post’s content.
All templates have access to a global site object variable: <code class="highlighter-rouge">site</code> as well as a page object variable: <code class="highlighter-rouge">page</code>.
The site variable holds all accessible content and metadata relative to the site.
The page variable holds accessible data for the given page or post being rendered at that point.</p>
<p><strong>Create a Template</strong>
Templates are created by properly formatting a file and placing it in the <code class="highlighter-rouge">_layouts</code> directory.</p>
<p><strong>Formatting</strong>
Templates should be coded in HTML and contain YAML Front Matter.
All templates can contain Liquid code to work with your site’s data.</p>
<p><strong>Rending Page/Post Content in a Template</strong>
There is a special variable in all templates named : <code class="highlighter-rouge">content</code>.
The <code class="highlighter-rouge">content</code> variable holds the page/post content including any sub-template content previously defined.
Render the content variable wherever you want your main content to be injected into your template:</p>
<pre><code>...
<body>
<div id="sidebar"> ... </div>
<div id="main">
{{content}}
</div>
</body>
...</code></pre>
<h3 id="sub-templates">Sub-Templates</h3>
<p>Sub-templates are exactly templates with the only difference being they
define another “root” layout/template within their YAML Front Matter.
This essentially means a template will render inside of another template.</p>
<h3 id="includes">Includes</h3>
<p>In Jekyll you can define include files by placing them in the <code class="highlighter-rouge">_includes</code> folder.
Includes are NOT templates, rather they are just code snippets that get included into templates.
In this way, you can treat the code inside includes as if it was native to the parent template.</p>
<p>Any valid template code may be used in includes.</p>
<h2 id="using-liquid-for-templating">Using Liquid for Templating</h2>
<p>Templating is perhaps the most confusing and frustrating part of Jekyll.
This is mainly due to the fact that Jekyll templates must use the Liquid Templating Language.</p>
<h3 id="what-is-liquid">What is Liquid?</h3>
<p><a href="https://github.com/Shopify/liquid">Liquid</a> is a secure templating language developed by <a href="http://shopify.com">Shopify</a>.
Liquid is designed for end-users to be able to execute logic within template files
without imposing any security risk on the hosting server.</p>
<p>Jekyll uses Liquid to generate the post content within the final page layout structure and as the primary interface for working with
your site and post/page data.</p>
<h3 id="why-do-we-have-to-use-liquid">Why Do We Have to Use Liquid?</h3>
<p>GitHub uses Jekyll to power <a href="http://pages.github.com/">GitHub Pages</a>.
GitHub cannot afford to run arbitrary code on their servers so they lock developers down via Liquid.</p>
<h3 id="liquid-is-not-programmer-friendly">Liquid is Not Programmer-Friendly.</h3>
<p>The short story is liquid is not real code and its not intended to execute real code.
The point being you can’t do jackshit in liquid that hasn’t been allowed explicitly by the implementation.
What’s more you can only access data-structures that have been explicitly passed to the template.</p>
<p>In Jekyll’s case it is not possible to alter what is passed to Liquid without hacking the gem or running custom plugins.
Both of which cannot be supported by GitHub Pages.</p>
<p>As a programmer - this is very frustrating.</p>
<p>But rather than look a gift horse in the mouth we are going to
suck it up and view it as an opportunity to work around limitations and adopt client-side solutions when possible.</p>
<p><strong>Aside</strong>
My personal stance is to not invest time trying to hack liquid. It’s really unnecessary
<em>from a programmer’s</em> perspective. That is to say if you have the ability to run custom plugins (i.e. run arbitrary ruby code)
you are better off sticking with ruby. Toward that end I’ve built <a href="http://github.com/plusjade/mustache-with-jekyll">Mustache-with-Jekyll</a></p>
<h2 id="static-assets">Static Assets</h2>
<p>Static assets are any file in the root or non-underscored subfolders that are not pages.
That is they have no valid YAML Front Matter and are thus not treated as Jekyll Pages.</p>
<p>Static assets should be used for images, css, and javascript files.</p>
<h2 id="how-jekyll-parses-files">How Jekyll Parses Files</h2>
<p>Remember Jekyll is a processing engine. There are two main types of parsing in Jekyll.</p>
<ul>
<li><strong>Content parsing.</strong>
This is done with textile or markdown.</li>
<li><strong>Template parsing.</strong>
This is done with the liquid templating language.</li>
</ul>
<p>And thus there are two main types of file formats needed for this parsing.</p>
<ul>
<li><strong>Post and Page files.</strong>
All content in Jekyll is either a post or a page so valid posts and pages are parsed with markdown or textile.</li>
<li><strong>Template files.</strong>
These files go in <code class="highlighter-rouge">_layouts</code> folder and contain your blogs <strong>templates</strong>. They should be made in HTML with the help of Liquid syntax.
Since include files are simply injected into templates they are essentially parsed as if they were native to the template.</li>
</ul>
<p><strong>Arbitrary files and folders.</strong>
Files that <em>are not</em> valid pages are treated as static content and pass through
Jekyll untouched and reside on your blog in the exact structure and format they originally existed in.</p>
<h3 id="formatting-files-for-parsing">Formatting Files for Parsing.</h3>
<p>We’ve outlined the need for valid formatting using <strong>YAML Front Matter</strong>.
Templates, posts, and pages all need to provide valid YAML Front Matter even if the Matter is empty.
This is the only way Jekyll knows you want the file processed.</p>
<p>YAML Front Matter must be prepended to the top of template/post/page files:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>---
layout: post
category : pages
tags : [how-to, jekyll]
---
... contents ...
</code></pre>
</div>
<p>Three hyphens on a new line start the Front-Matter block and three hyphens on a new line end the block.
The data inside the block must be valid YAML.</p>
<p>Configuration parameters for YAML Front-Matter is outlined here:
<a href="https://github.com/mojombo/jekyll/wiki/YAML-Front-Matter">A comprehensive explanation of YAML Front Matter</a></p>
<h4 id="defining-layouts-for-posts-and-templates-parsing">Defining Layouts for Posts and Templates Parsing.</h4>
<p>The <code class="highlighter-rouge">layout</code> parameter in the YAML Front Matter defines the template file for which the given post or template should be injected into.
If a template file specifies its own layout, it is effectively being used as a <code class="highlighter-rouge">sub-template.</code>
That is to say loading a post file into a template file that refers to another template file with work in the way you’d expect; as a nested sub-template.</p>
<h2 id="how-jekyll-generates-the-final-static-files">How Jekyll Generates the Final Static Files.</h2>
<p>Ultimately, Jekyll’s job is to generate a static representation of your website.
The following is an outline of how that’s done:</p>
<ol>
<li>
<p><strong>Jekyll collects data.</strong>
Jekyll scans the posts directory and collects all posts files as post objects. It then scans the layout assets and collects those and finally scans other directories in search of pages.</p>
</li>
<li>
<p><strong>Jekyll computes data.</strong>
Jekyll takes these objects, computes metadata (permalinks, tags, categories, titles, dates) from them and constructs one
big <code class="highlighter-rouge">site</code> object that holds all the posts, pages, layouts, and respective metadata.
At this stage your site is one big computed ruby object.</p>
</li>
<li>
<p><strong>Jekyll liquifies posts and templates.</strong>
Next jekyll loops through each post file and converts (through markdown or textile) and <strong>liquifies</strong> the post inside of its respective layout(s).
Once the post is parsed and liquified inside the the proper layout structure, the layout itself is “liquified”.
<strong>Liquification</strong> is defined as follows: Jekyll initiates a Liquid template, and passes a simpler hash representation of the ruby site object as well as a simpler
hash representation of the ruby post object. These simplified data structures are what you have access to in the templates.</p>
</li>
<li>
<p><strong>Jekyll generates output.</strong>
Finally the liquid templates are “rendered”, thereby processing any liquid syntax provided in the templates
and saving the final, static representation of the file.</p>
</li>
</ol>
<p><strong>Notes.</strong>
Because Jekyll computes the entire site in one fell swoop, each template is given access to
a global <code class="highlighter-rouge">site</code> hash that contains useful data. It is this data that you’ll iterate through and format
using the Liquid tags and filters in order to render it onto a given page.</p>
<p>Remember, in Jekyll you are an end-user. Your API has only two components:</p>
<ol>
<li>The manner in which you setup your directory.</li>
<li>The liquid syntax and variables passed into the liquid templates.</li>
</ol>
<p>All the data objects available to you in the templates via Liquid are outlined in the <strong>API Section</strong> of Jekyll-Bootstrap.
You can also read the original documentation here: <a href="https://github.com/mojombo/jekyll/wiki/Template-Data">https://github.com/mojombo/jekyll/wiki/Template-Data</a></p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope this paints a clearer picture of what Jekyll is doing and why it works the way it does.
As noted, our main programming constraint is the fact that our API is limited to what is accessible via Liquid and Liquid only.</p>
<p>Jekyll-bootstrap is intended to provide helper methods and strategies aimed at making it more intuitive and easier to work with Jekyll =)</p>
<p><strong>Thank you</strong> for reading this far.</p>
<h2 id="next-steps">Next Steps</h2>
<p>Please take a look at <a href=""></a>
or jump right into <a href="">Usage</a> if you’d like.</p>