Adding or Updating Commands

1. Introduction

Commands describe actions that can be initiated by a n3phele user to run on a cloud. Commands can be private so that only you can see and run them, or commands can be made public so that any n3phele user can see and run them. Commands are versioned. This means you can have multiple versions of command available simultaneously in n3phele, for example a stable version that users will see by default, as well as experimental version that you are currently developing. 

 2. Anatomy of a command

 n3phele commands are defined in 2 parts in a description file that you can create with a text editor. The first part is the interface definition that describes aspects such as the command name and version, the command input and output files, and the command parameters. The second part defines one or more execution profiles that describe how the command is run on a particular cloud. You can have multiple execution profiles for a command on a cloud, for example a small (cheap) and large (costlier) footprint execution environment. 
 
The description file is in JSON format. JSON format is fundamentally structured data in the form of name value pairs. The best way to create a command is to download and start editing one of the templates in table 1.
 

 3. Grep command example

To illustrate, we work through a simple example creating a n3phele grep command. Later on we will describe all the fields in detail, but for now this is just a simple example to get started. For the grep command we want to supply an input file and a search string, and have a result file that will contain all the lines from the input file that match that search string.
 
To start, download the 1 input file, 1 output file, 1 parameter template from table 1.

 3.1 Grep command interface definition portion

Open it in your favorite text editor, and you will see something similar to figure 1 which is showing the interface definition portion of the file. The text in this portion of the definition will show up in the n3phele window when you select the command.  
 
  1. To start, we will define the command name. To do this we will do a "replace all" command in the editor, to replace "mycommand" with "grep". There are multiple instances of mycommand in the file, so do use a replace all.
  2. Next you want to update the description text on line 5 to "print lines matching a pattern"
  3. Update the command version number on line 8 to "1.0RC"
  4. Update the input file1 description on line 12 "file to search"
  5. Update the output file 1 description on line 17 to "match results"
  6. Change the paramater name on line 21 from param1 to pattern
  7. Add some description text on line 22 "file search pattern"
 
After you complete this, you will see similar to figure 2 in your editor window

 3.2 Grep command execution profile 

Scrolling down the file shows the execution profile, as shown in figure 3. 
Execution profiles describe how to activate a command in a particular cloud environment. Execution profiles are basically a list of actions that need to be executed in order to run the command. In the shown file there are two actions, the first action (lines 32-65) describes how to set up the virtual machine. The 2nd action (lines 76-104) describes running a shell command that produces and consumes some files. In particular, the shell action, has an input and output files with names the match those specified in the command interface definition discussed in 3.1 namely "input1" and "output1".
 
Lets look at the input file more closely. On line 75 we define the filename for this input file to be input.txt. When the command is run, n3phele will take the user nominated file and transfer it to the environment in which the shell command is running giving it a final filename of "input.txt". Similarly, after the shell command has completed, n3phele will look for a file called "output.txt" (see line 102) and will transfer that file to the destination that was nominated on the n3phele command line. Any command that we run will refer to the input and output files using these specified local filenames.
 
We now make the following updates to implement our greg command exection profile:
 
Note lines 33, 34, 69 and 80 were changed by the replace all action in step 1 of 3.1.
  1. Update the execution profile description in line 28 to "on-demand m1.small"
  2. next we will update line 82 which defines the linux shell command that we want to run. First we layout our shell command, and then we will have to pay some attention to quoting. The shell command we want is:
            grep nominated_pattern input.txt >output.txt
 
        where nominated_pattern is the parameter that was specified on the command line.
 
To create this command line we will write an expression that combines constant text with the variable that holds the parameter value, as follows:
 
            "grep "+$<pattern>+" input.txt >output.txt"
 
The construct $<x> returns the value of the command line parameter named "x". In line 21 we defined a parameter named "pattern".  Similarly, the construct $<a.x> returns the parameter named "x" from the action "a". (See an example on line 87). In general n3phele value and defaultValue fields are expressions of arbitrary complexity, including c language conditionals of
 
            logical_expression ? execute_when_true : execute_when_false
 
which is very useful in constructing conditional command lines.
 
Because JSON also encloses values in quotes, we need to transform the representation of our desired command line by escaping the double quotes with a backslash. So our command line in JSON now looks like:
 
           "\"grep \"+$<pattern>+\" input.txt >output.txt\""
 
Now, suppose we wanted to surround the pattern in the shell command with quotes. Ignoring JSON, the expression would be:
 
        "grep \""+$<pattern>+"\" input.txt > output.txt"
 
showing that we would need to quote the double-quote character that we are using inside the constant strings. Now, if we transform this into the JSON form we need to escape both the double quotes (i.e. " becomes \")  as well as the backslash (i.e. \ becomes \\), which gives us:
 
          "\"grep \\\"\" +$<pattern> +\"\\\"  input.txt >output.txt\""
 
This is what we need to paste into line 82 replacing "\" echo hello; cat <input.txt >output.txt \"".
 
If you want to check your edits, a complete edited form of the tutorial can be downloaded here.
 

4. Installing the Grep command 

 To install your command log into n3phele at https://n3phele.appspot.com
 
 
 At the top right of the window click on the link "create a new command".
 
A Command Definition Import popup appears. Click on the "Browse.." button to select the definition file to be imported. Once selected use the "import" button to complete the definition.
 
The definition is then processed, and if no errors occur you will get a confirmation message about the creation of the command and associated profile including information on the uri that has been created. It is strongly recommended that you paste the returned URI string into the command definition file (see line 3 of Figure 2), as this will facilitate future command updates.
 

5. Dealing with Errors 

Command import can fail for two main reasons:
 
  • The specified command name is already in use by someone else.
    In this case the import will return with a message "ignored: already exists". You can use the Command window to search for that command to understand what that existing command does and evaluate if you need to create your command. If you still want to create the command, then change the name specified on line 4 of figure 2.
  • The import JSON file is badly formed
    There are a number of web sites that can help format and validate JSON files. Google JSON validator and find your favorite. http://jsonlint.com/ is one example.

6. Update a Command

 If you want to update your command, ensure that you have specified a URI on line 3 of figure 2, and import the definition file again using the "create a new command" link on the command page.

7. Delete a command

There is no explicit way to delete a command that you have created. Eventually garbage collection will remove old never used commands. Email n3phele@gmail.com is you wish to accelerate that process. 

8. Other examples

Here is another contributed example runPythonOnQiime. This template runs the amazon Qiime AMI -which has a rich set of python libraries available- and then executes a python script that you have authored on it with your data. This is great if you have a couple of simple scripts that you use in conjunction with QIIME. Take a look at line 76

 wget -q -O - https://MYBUCKET.s3.amazonaws.com/myscript.py > myscript.py; python myscript.py input.txt output.txt

This line downloads a python script called myscript.py from an amazon S3 buckey "MYBUCKET" (make sure the permissons are public read on that file), and then runs it with two arguments the input and output files to be processed.

 
Table 1. n3phele command templates
 
Input FIles  Output Files  Parameters   
 download
 download 
download 
   
 
Figure 1. Simple n3phele command definition 
 

1 {
2    "command" : {
3        "uri" : "",
4        "name" : "mycommand",
5        "description" : "insert description here",
6        "public" : "false",
7        "preferred" : "false",
8        "version" : "0.0",
9        "icon" : https://www.n3phele.com/icons/custom,
10       "inputFiles" : {
11       "name" : "input1",
12       "description" : "Input file1",
13        "optional" : "false"
14     },
15     "outputFiles" : {
16         "name" : "output1",
17         "description" : "output file1",
18          "optional" : "false"
19     },
20     "executionParameters" : {
21        "name" : "param1",
22         "description" : "insert parameter description here",
23         "type" : "String",
24         "defaultValue" : ""
25     },

    ...... 

 
Figure 2. Simple n3phele command definition 

1 {
2    "command" : {
3        "uri" : "",
4        "name" : "grep",
5        "description" : "print lines matching a pattern",
6        "public" : "false",
7        "preferred" : "false",
8        "version" : "1.0RC",
9        "icon" : https://www.n3phele.com/icons/custom,
10       "inputFiles" : {
11       "name" : "input1",
12       "description" : "file to search",
13        "optional" : "false"
14     },
15     "outputFiles" : {
16         "name" : "output1",
17         "description" : "match results",
18          "optional" : "false"
19     },
20     "executionParameters" : {
21        "name" : "pattern",
22         "description" : "file search pattern",
23         "type" : "String",
24         "defaultValue" : ""
25     },

    ...... 

 
 
 
 
Figure 3. Simple n3phele command definition 

 26    "executionProfiles" : {
 27        "id" : "1",
 28        "description" : "execution profile description",
 29        "cloud" : https://n3phele.appspot.com/resources/cloud/586001,
 30        "public" : "true",
 31        "actions" : [{
 32                "name" : "vm",
 33                "description" : "grep virtual machine",
 34                "workloadKey" : "grep",
 35                "action" : http://www.n3phele.com/createvm,
 36                "inputParameters" : [{
 37                "name" : "imageId",
 38                "description" : "Virtual machine image identifier",
 39                "type" : "String",
 40                "value" : "ami-8c1fece5"
 41            },
 42            {
 43                "name" : "minCount",
 44                "description" : "instance count",
 45                "type" : "String",
 46                "value" : "1"
 47            },
 48            {
 49                "name" : "instanceType",
 50                "description" : "Virtual machine size",
 51                "type" : "String",
 52                "value" : "m1.small"
 53            },
 54            {
 55                "name" : "securityGroups",
 56                "description" : "Security group specifies the open TCP/IP ports for the VM",
 57                "type" : "String",
 58                "value" : "n3phele-default"
 59            },
 60            {
 61                "name" : "userData",
 62                "description" : "Specifies initialization commands for the VM.",
 63               "type" : "String",
 64                "value" : "#!/bin/bash\necho starting injection... \nset -x\nwget -q -O -  https://n3phele-agent.s3.amazonaws.com/n3ph-install-tgz-basic | su - -c '/bin/bah -s ec2-user ~/agent ~/sandbox' ec2-user\n"
 65            }]
 66        },
 67        {
 68            "name" : "shell",
 69            "description" : "grep shell",
 70            "workloadKey" : "shell",
 71            "action" : http://www.n3phele.com/executecommand,
 72            "inputFiles" : {
 73                "name" : "input1",
 74                "description" : "Input file1",
 75                "filename" : "input.txt",
 76               "optional" : "false"
 77            },
 78            "inputParameters" : [{
 79                "name" : "command",
 80                "description" : "grep command",
 81                "type" : "String",
 82                "value" : "\" echo hello; cat <input.txt >output.txt \""
 83            },
 84            {
 85                "name" : "agentBaseUrl",
 86                "type" : "String",
 87                "value" : "$<vm.agentBaseUrl>"
 88            },
 89            {
 90                "name" : "agentUser",
 91               "type" : "String",
 92                "value" : "$<vm.agentUser>"
 93            },
 94            {
 95                "name" : "agentPassword",
 96                "type" : "String",
 97                "value" : "$<vm.agentPassword>"
 98            }],
 99            "outputFiles" : {
100                "name" : "output1",
101                "description" : "output file1",
102                "filename" : "output.txt",
103                "optional" : "false"
104            }
105        }]
106    }]
107 }

 
 
 
 
 
 
 
 
 
 
Subpages (1): samples
Comments