1. IntroductionCommands describe actions that can be initiated by a n3phele user to run on a cloud. Commands can be private so that only you can see and run them, or commands can be made public so that any n3phele user can see and run them. Commands are versioned. This means you can have multiple versions of command available simultaneously in n3phele, for example a stable version that users will see by default, as well as experimental version that you are currently developing.
2. Anatomy of a command n3phele commands are defined in 2 parts in a description file that you can create with a text editor. The first part is the interface definition that describes aspects such as the command name and version, the command input and output files, and the command parameters. The second part defines one or more execution profiles that describe how the command is run on a particular cloud. You can have multiple execution profiles for a command on a cloud, for example a small (cheap) and large (costlier) footprint execution environment.
The description file is in JSON format. JSON format is fundamentally structured data in the form of name value pairs. The best way to create a command is to download and start editing one of the templates in table 1.
To illustrate, we work through a simple example creating a n3phele grep command. Later on we will describe all the fields in detail, but for now this is just a simple example to get started. For the grep command we want to supply an input file and a search string, and have a result file that will contain all the lines from the input file that match that search string.
To start, download the 1 input file, 1 output file, 1 parameter template from table 1.
Open it in your favorite text editor, and you will see something similar to figure 1 which is showing the interface definition portion of the file. The text in this portion of the definition will show up in the n3phele window when you select the command.
After you complete this, you will see similar to figure 2 in your editor window
3.2 Grep command execution profileScrolling down the file shows the execution profile, as shown in figure 3.
Execution profiles describe how to activate a command in a particular cloud environment. Execution profiles are basically a list of actions that need to be executed in order to run the command. In the shown file there are two actions, the first action (lines 32-65) describes how to set up the virtual machine. The 2nd action (lines 76-104) describes running a shell command that produces and consumes some files. In particular, the shell action, has an input and output files with names the match those specified in the command interface definition discussed in 3.1 namely "input1" and "output1".
Lets look at the input file more closely. On line 75 we define the filename for this input file to be input.txt. When the command is run, n3phele will take the user nominated file and transfer it to the environment in which the shell command is running giving it a final filename of "input.txt". Similarly, after the shell command has completed, n3phele will look for a file called "output.txt" (see line 102) and will transfer that file to the destination that was nominated on the n3phele command line. Any command that we run will refer to the input and output files using these specified local filenames.
We now make the following updates to implement our greg command exection profile:
Note lines 33, 34, 69 and 80 were changed by the replace all action in step 1 of 3.1.
grep nominated_pattern input.txt >output.txt
where nominated_pattern is the parameter that was specified on the command line.
To create this command line we will write an expression that combines constant text with the variable that holds the parameter value, as follows:
"grep "+$<pattern>+" input.txt >output.txt"
The construct $<x> returns the value of the command line parameter named "x". In line 21 we defined a parameter named "pattern". Similarly, the construct $<a.x> returns the parameter named "x" from the action "a". (See an example on line 87). In general n3phele value and defaultValue fields are expressions of arbitrary complexity, including c language conditionals of
logical_expression ? execute_when_true : execute_when_false which is very useful in constructing conditional command lines.
Because JSON also encloses values in quotes, we need to transform the representation of our desired command line by escaping the double quotes with a backslash. So our command line in JSON now looks like:
"\"grep \"+$<pattern>+\" input.txt >output.txt\""
Now, suppose we wanted to surround the pattern in the shell command with quotes. Ignoring JSON, the expression would be:
"grep \""+$<pattern>+"\" input.txt > output.txt"
showing that we would need to quote the double-quote character that we are using inside the constant strings. Now, if we transform this into the JSON form we need to escape both the double quotes (i.e. " becomes \") as well as the backslash (i.e. \ becomes \\), which gives us:
"\"grep \\\"\" +$<pattern> +\"\\\" input.txt >output.txt\""
This is what we need to paste into line 82 replacing "\" echo hello; cat <input.txt >output.txt \"".
If you want to check your edits, a complete edited form of the tutorial can be downloaded here.
4. Installing the Grep command To install your command log into n3phele at https://n3phele.appspot.com
At the top right of the window click on the link "create a new command".
A Command Definition Import popup appears. Click on the "Browse.." button to select the definition file to be imported. Once selected use the "import" button to complete the definition.
The definition is then processed, and if no errors occur you will get a confirmation message about the creation of the command and associated profile including information on the uri that has been created. It is strongly recommended that you paste the returned URI string into the command definition file (see line 3 of Figure 2), as this will facilitate future command updates.
Command import can fail for two main reasons:
6. Update a CommandIf you want to update your command, ensure that you have specified a URI on line 3 of figure 2, and import the definition file again using the "create a new command" link on the command page. 7. Delete a commandThere is no explicit way to delete a command that you have created. Eventually garbage collection will remove old never used commands. Email n3phele@gmail.com is you wish to accelerate that process. 8. Other examplesHere is another contributed example runPythonOnQiime. This template runs the amazon Qiime AMI -which has a rich set of python libraries available- and then executes a python script that you have authored on it with your data. This is great if you have a couple of simple scripts that you use in conjunction with QIIME. Take a look at line 76 wget -q -O - https://MYBUCKET.s3.amazonaws.com/myscript.py > myscript.py; python myscript.py input.txt output.txt This line downloads a python script called myscript.py from an amazon S3 buckey "MYBUCKET" (make sure the permissons are public read on that file), and then runs it with two arguments the input and output files to be processed. |
Table 1. n3phele command templates
Figure 1. Simple n3phele command definition
Figure 2. Simple n3phele command definition
Figure 3. Simple n3phele command definition
|
