Linux is only free if your time has no value
What began at first as a quick search into how one might tackle a slightly inconvenient issue turned into an entire Saturday afternoon researching the intracacies of bash, argument and variable parsing and newline conventions, all for the pursuit of the perfect ideal one liner.
Given that I’m upto week 4 of my Coder Academy course and we’ve now moved onto HTML and CSS it seemed fitting to relay my command line struggles by practicing what I’ve been learning in the form of my first blog post.
The story begins with the back and forth messaging on Slack with fellow classmate John Voon. We’re practicing GPG encryption.
We get to the point of exchanging keys, which we just post through Slack. Admittedly what follows could have been summed up in a very short “got this problem? Here’s how you deal with that” sort of post but then that wouldn’t illustrate the numerous different attempts it took me before I finally came to reach that solution (were you after the TL:DR?).
I need to clarify here the distinction between PGP and GPG, PGP is the standard and GPG is a program that implements the PGP standard, as such the actual text that denotes either a colleagues public key or the cipher text of a message I will be referring to as PGP text.
Importing a public key is as easy as typing the following into your linux terminal then pasting in the PGP text:
When no other arguments are provided the program expects to be given the text of the public key through stdin. This is what I want, because I want to just copy the text straight out of Slack and paste into the terminal, thereby removing the need to deal with files that would only be temporary anyway.
I go ahead and paste the text in, the last few lines coupled with the result looks like this:
Notice the spaces between the lines as well as the concerning error messages directly afterwards?
“CRC error”
In other cases it may appear as “invalid radix64 character”.
For example:
It’s obvious what’s happened, the annoying extra newlines are screwing up the parsing of the PGP block.
I’m used to this because of the same annoyance when attempting to copy posted code from Slack during coding classes into my IDE which also results in extra newlines that historically I’ve been removing by hand since it was usually just 3 or 5 lines worth.
After copying the PGP text straight from Slack and pasting into the terminal failed, I figured perhaps pasting first into gedit (a very simple graphical text editor that ships with Ubuntu, like Notepad on Windows) and then copying from there into the terminal might fair better (probably from the simple trick you can do in Microsoft Excel to quickly convert date values into text format without using excel formulas).
So I copy from Slack and paste into gedit. The new lines appear and looks like what was pasted into the terminal. That won’t work.
So next I try to download a copy of the text snippet using the download option instead of just highlighting the text in my browser. I opt to open it straight with gedit rather than saving as a new file on the computer.
It opens in gedit and it now looks good, it doesn’t have all the extra newlines.
Great! I think.
I highlight the contents of gedit, copy it, and paste into GPG’s input in the terminal.
Same problem. That’s weird, I save the gedit file as an actual file then run:
It works now and the key is imported succesfully.
Someone else might at this point have just accepted that that is the way to do it and carried on enjoying their Saturday, but I felt defeated. Why did I have to do that?? Seeing that as just one extra step means forgetting about the step where you have to delete the file later on!
I kept trying to find out what the difference between copying the text from a file could be versus saving the same exact file and then referring to the file.
I’m suspicious that there must be some extra newlines hiding in the text that gedit is just not showing or acting on and that’s why when in gedit it looks normal but when pasted into the terminal there appears multiple new newlines.
In gedit I set the cursor to the very end of one of the lines. I hit delete half expecting for no apparent change to occur, imagining that if a hidden newline character was present it would be removed without any apparent change in appearance. But no, the lower line is brought up to the cursors position, it would appear that there was indeed only one new line character there after all, and hitting delete removed it so that now there is no newline at all and just one very long line of text.
I hit enter to put the newline back in there. The text block now looks the same as it did before.
Maybe something’s changed and there are no longer any extra newlines?
I copy it all again and attempt to paste into GPG again in the terminal.
Same thing, same problem. Extra new lines are being added.
I repeat this a few times because I’m insane and expect something different to happen before I scroll up to see the pasted text that was pushed out of sight. I notice that the lines are different at the top. It seems my action of deleting a newline and then immediately and pointlessly replacing it again within gedit actually must have removed some sort of hidden characters because the lines that I did that to now appear in the terminal as they were meant to in the first place.
Can gedit be trusted? gedit’s hiding something from me. I do some searching online and find that there’s a plugin you can get for gedit called draw spaces.
I install it with the promise that it will show to me previously invisible control characters.
When I run it with that I only see single new line characters:
The ones I readded appear to be displayed the same as the ones I haven’t touched. Yet upon pasting into the terminal, the difference between them and the lines I hadn’t touched is obvious:
There’s definitely some new lines hidden there but I can’t see them and I want to.
Introducing the hex editor. My suspicions are confirmed when I enlist the powers of Bless the hex editor, the bytes at the end of each line look like an x0D followed by an x0A, but the lines where I replaced the newlines now just have a single x0A.
see this snippet:
checking online resources reveals:
x0D = carriage return character
x0A = line feed character
The line feed character is the Unix way of representing a new line.
The carriage return character is the Mac/OSX way of representing a new line.
The combination of carriage return and line feed characters is the way Windows machines represent new lines.
Apparently that is also the way the text in Slack represents new lines as well.
Now, how do we get rid of them.
Searching reveals some options. I can use the sed command, or I can use the tr command. Both have the ability to delete characters from a stream or file of text. Maybe there is a useful way to employ those commands when I paste in the GPG text block into the terminal to get them to clean it of carriage return characters before sending it into the GPG program?
In order to practice with the commands, I want to have the PGP text in a variable to test it out:
I can verify the text (along with the extra newlines) is in the variable:
I’ve buffered the variable with double quote literals because the following gave a printf error:
This was due to the start of the text in $testvar being the start of the PGP text:
-----BEGIN PGP MESSAGE-----
In other words, calling
is the same as calling:
but we want it to be:
so that’s why I tried this:
but unfortunately that resulted in only the first ‘word’ being printed:
The solution was to wrap it all again with double quotes:
I got caught out because what was happening was that when I was pasting the PGP text from Slack into the terminal, the terminal was swapping out the carriage return \r characters for line feed \n characters, as such any attempts to run the text through a filter first to remove all \r characters had no effect, but I didn’t realise that at the time:
results in no change to the text, however:
results in all newlines being removed.
I figured it out when I pasted the text into Bless and noticed that the newline character combinations were now x0A x0A not x0D x0A like it is when the text is not copied out of the browser straight from Slack but rather taken from the file that is downloaded when you opt to download the snippet from Slack instead.
It’s for this reason that I switched to the sed command for removing extra empty newlines:
I found the above with a search online for a sed command to remove blank lines.
Here it is being tested against the contents of the $testvar variable:
which worked to fix up the formatting of the text, the one liner with the variable then became:
Note the extra newline inserted before the $testvar variable? That’s because the absence of that results in the PGP header line beginning with a double quote character which is apparently not valid for a PGP header, but we can’t take away the double quote because then the printf command will fail. The following results in a GPG error
because the header is now this:
"-----BEGIN PGP MESSAGE-----
instead of this:
-----BEGIN PGP MESSAGE-----
I’m using printf instead of echo because it can print multiple lines whereas echo will not.
Telling printf to print a string that consists of the double quotes surrounding the $testvar variable was necessary before because otherwise the expansion of the $testvar to what its value is would remove the outer double quotes, and then printf would then see the value as not the string to print, but part of some optional arguments being passed to it. However as you will see below, the addition of the intial newline character means the explicit double quotes are no longer necessary.
The next evolution was to do away with the use of any variable at all and just paste straight in:
Unfortunately this means that the large block of PGP text is now considered part of the command you just entered, which means it forms a part of your bash history and subsequent attempts to try and cycle through previously entered commands can sometimes result in the terminal’s text display getting a bit screwed up! It also means that we’re not running a command and then pasting in our PGP text, we’re typing the first half of the command, pasting our text, and then typing the rest of the command.
At this point I felt the easiest workaround would involve pasting the copied text into gedit and just using gedit’s find and replace function to remove the superfluous newline characters. That’s when I discovered a difference of behaviour between simply copying the text and pasting into gedit vs using the slack channel snippet’s download button to download the snippet and open with gedit.
The former will result in the carriage return characters being replaced by line feed characters whereas the latter will retain the carriage return characters.
This means when pasting copied text into gedit, my find and replace will be:
find: \n\n
replace with: \n
However when downloading the entire plain text snippet and opening with gedit my find and replace will be:
find: \r
replace with: <nothing>
So based on the results from pasting directly into the terminal and then having there be no \r characters to remove (hence the sed command to remove \r characters having no effect but removing superfluous newlines did), and the same experience with pasting straight into gedit, I’m guessing Unix programs just automatically convert carriage return characters to line feed characters.
It was during the writing of this post that I went searching for a way to do something related in the terminal when I read about the read command.
Such a command means we can revisit the earlier one liner but amend it to use read to store the pasted PGP text into a variable which is then filtered and passed to GPG! This means the PGP text itself does not form part of the command itself. Which means the command will now be the same for all different PGP texts we might use in future, which means we can store it as an alias declared in our .bashrc file so that we don’t even have to type it all out every time!
Double win!
But there is a problem. The read command does not really handle multi-line strings too well, which is what we’d be pasting in.
Some more searching later and I find this option:
It only requires that I type CTRL+D after pasting as opposed to simply enter.
With that all typed, and without any further ado, I give you the final version of the most elegant way I’ve found so far to be able to just paste PGP text straight into a Linux terminal, have it piped through programs to filter it of any extra carriage returns and then pass that resulting text straight into GPG (all one line):
The above can be added to your .bashrc file to save you from typing it out each time, the following two lines added to the end of your .bashrc file means when you want to import a friends public key and you’ve copied the text, in your terminal you just have to write gpgkey and if you’re pasting in a message to be decrypted you just have to write gpgmsg:
Remember that your .bashrc is only read by non-authenticating terminals (the ones you open from your desktop) when they start, so once you make the above changes they won’t work in any terminals that were already open.
The only thing right now that is not quite perfect is that there is naturally a blank line in the PGP text immediately following the header. This line is also removed by the above code which doesn’t discriminate between repeats of line feed characters. As such while the process still works and the message is successfully decrypted, GPG throws an error:
In the following, note that the highlighted section is the same for both, but that the first match after that differs between them.
This means that when the text is simply copied straight out of the browser and pasted into the terminal, which is the most convenient method, the distinction between the true blank line and the introduced ones has been removed:
This means there will be no way to determine where there used to be naturally a blank line, the only way I can think of to put that line back in there is to tell the command to not remove the second blank line, or to put it back in afterwards which just seems a bit hard-codey..
Given that the solution works as I intended I’m happy at this point to just put up with the output of that warning message from GPG about an invalid armour header.