Bash Scripting for Boffins (Article 1)

Boffin Defined

A Boffin is an early 20th century, British colloquialism for a person engaged in scientific or technical research or related work. This fairly describes me for the last 25 years. I would further extend that definition to include the broad swath of educated masses compelled to make technical solutions for processes which have never previously been technical in nature, which includes about everyone except for management and administration. The very fact that you are reading this article tells me, that you are likely a fellow traveler and a Boffin.

As Boffins, we are not the 4.0 student, medical doctor, or Mensa members. We are the solid B students, with more innovation and less focus than would otherwise be required to go the full-monty. Aldous Huxley would have pegged us as Betas because we're harder working than those ivory towered, Alpha blow-hards while being intellectually superior to those Gammas and Deltas. We solve problems in a more elegant and resourceful manner than our Alpha superiors, but... we needed money for beer, not grad school!

BASH Defined

BASH scripting stands for Bourne Again SHell (BASH), named after Dr. Bourne who wrote the Bourne Shell for Bell Labs, by Brian Fox for the GNU project. BASH is a soft typed, functional scripting language that is specifically designed to interact with the kernel and file system in Linux. Most Gnu Linux projects have text based handles which allow them to accept and produce data to and from the command shell. This gives us several prototyping advantages including:

• Your interface does not need to build arrays in order to manage process state, because the kernel can do that for you. You just need to query the kernel (i.e, via the PS utility) to control child processes.
• The EXT3 files system is at your disposal and can be used as a very efficient, reliable storage mechanism allowing multiple processes to concurrently access the same file. By contrast, in power shell (for instance), you are stuck with NTFS.
• You can build your own ingesters (say for mysql) as opposed to relying on "load" commands from MySQL (for instance)
• You can build libraries (of sorts), loading pre-written scripts to help write subsequent functions.
• Bash handles memory management, making your applications deceptively stable.
• You can pipe data into, out of, and between applications using the above facilities.
• Because your prototype is run-time interpreted, you can literally debug and develop as you go.

If you are thinking to yourself "Look, I know X language...For the love of God why do you want us to look at BASH!?", then read on.

Why should Boffins Look at BASH?

Simply put: Rapid, Robust Prototyping and Automation

As Boffins, we are typically on the front lines, without the budget, with the deadline, with service levels to meet, requiring a tailored technical solution which does not yet exist. We beg and plead with our Alpha handlers for money, which sometimes they allow us. After being allowed to hire a resource, we are told that they can't or won't do it that way, or that the one feature your staff absolutely needed was technically not in your specifications. Unfortunately, those are the specifications which, more often than not, we gave them...the ones for a business or technical process which does not yet exist, using a modification to a dated manual procedure, relying on feedback from non-technical people ...who all want it to be simple, "look like Google", and ...it should do their job for them.

Some of us will crack open Python or Java, and try to develop something approaching what we need. Typically when doing 'real coding' however you are limited by the libraries in the languages that are available. Then there are the hazards of making a mistake which compromises data quality or business processes. Bash, however, allows us to stand on the shoulders of giants, well vetted projects in order to accomplish what we need.

The number of full scale, well coded projects and solutions, freely available for use in Linux, and typically controllable from the Linux shell, number in the thousands. That's just including those already available in package managers. If you include those that you will need to compile, who knows!

Ad arguendo, let's assume there are only 1000 programs we can run on our Ubuntu Linux distribution, and that we will need to incorporate the functionality of 10(?) different programs in order to provide a working technical solution, then there are a huge number of solutions available. In an optimistic sense, the approximate number of unique combinations of ten programs available to solve any problem could be expressed as...

Suffice it to say...we have pretty good odds of solving any one problem using Bash to combine functionality from other, already constructed programs.

My Premise

BASH is (bar none) the absolute best prototyping solution for the underfunded Boffin. It will enable you to make simple and robust prototypes which accomplish the following goals:

• Immediately meet any technical need (for free)
• Determine the true requirements for your project
• Provide crystal clear technical specifications for any larger project and a more permanent solution

Let's consider for a moment that your boss needs you to obtain minute data from a web site each hour, and calculate each day some summary statistics on it, produce a real time cumulative graph of the data, and email it to him. You can easily do this with simple shell scripts and calling the right binaries.

My Challenge for This series

In the next few posts I will set up a test environment using virtual box running on a windows machine, using Ubuntu as a prototyping work station on Windows 10. My challenge is to solicit requests for processes which someone may actually need to prototype in their work environment. Using these requests I will offer solutions, teaching bash and the overall methodology described above along the way.

DISCLAIMER There are many Unix admins which are better at script foo than I. What I bring to the table, however, is a Rube Goldberg approach to knitting disparate applications together in order to automate or prototype anything in a robust manner.

Here are a few example problems to give you an idea of the types of problems I want you, the reader, to suggest:

Reader Problem: Scrape a web site for some text, query a serial port, obtain a modbus register, and parse or store the value we want. Aggregate, tally, and statistically summarize the value over time. Email my boss a graph and a spreadsheet on a weekly basis.
Solution Summary: BASH([wget,sjinn,modbuspoll] ->ASCII-> awk ->Mutt)

Reader Problem :Aggregate three sets of data and generate an interactive 3 dimensional graph from the data. Compress and encrypt the data and the image, and store them on a google drive.
Solution Summary: BASH([cat,sort,grep,awk] ->ASCII-> Gnuplot ->Grive)

Reader Problem: Query a device over modbus remotely, and send results daily via email
Solution Summary: Gnome + VNC Server + Putty + IPTables + BASH(modbuspoll->ASCII->Mutt)

....next article (Bash Scripting for Boffins 2: Setting up an Ubuntu Desktop in Virtual Box on Windows 10).

Please suggest problems in the feedback below.