RPG and YAJL, as Well as DATA-INTO News

This month, I’ll give you a quick update about RPG’s new DATA-INTO opcode and then focus on the basics of how you can host your own custom REST web service API written in RPG with the help of the standard IBM HTTP Server (powered by Apache) and the YAJL JSON library.

DATA-INTO News

IBM has posted documentation online for the new DATA-INTO opcode, and they promise that PTFs will be available on March 19th, 2018. That means that by the time this blog entry goes live, it’ll be available for you to install and try.

Unfortunately, I’m writing this before the PTFs are available, so I haven’t been able to try it yet. I hope to do so in time for next month’s blog.

Providing a Web Service with RPG and YAJL

I’ve been creating quite a few JSON-based REST APIs using the YAJL toolkit lately, and with DATA-INTO on the horizon, I thought it’d be a good time to give you an example. If you’re not familiar with YAJL, it is a free tool for reading and writing JSON documents. Since YAJL is written in C, I was able to compile it as a native ILE C service program and create an RPG front-end for it that makes it both easy to use and extremely efficient.

Of course, IBM provides us with the wonderful Integrated Web Services (IWS) tool that can do both REST and SOAP APIs using both JSON and XML. The advantage that IWS has over YAJL is that IWS does most of the work for you, simplifying things tremendously. On the other hand, YAJL is faster and lets you deal with much more complex documents. For the work that I do, I usually find YAJL works better.

Prerequisites

Before you can use the techniques that I describe in this post, you’ll need to have the following items installed:

  • IBM HTTP Server (powered by Apache)
    • This is available on the OS installation media as licensed program option 57xx-DG1 – you’ll want to see the IBM Knowledge Center for documentation about how to install it and its prerequisites
  • YAJL (Yet Another JSON Library)
  • ILE RPG compiler at version 6.1 or newer

Initial Setup

I recommend that you create a new library in which to put your programs that implement REST APIs. It’s certainly possible to put them in an existing library, but using a separate one helps with security. If you set up the HTTP server to only access the new library, you won’t need to worry about someone trying to use your server to run programs that they shouldn’t.

In my example, I will create a library named YAJLREST.

CRTLIB LIB(YAJLREST)

You’ll also need to set up the IBM HTTP server to point to your new library. This is best done using the HTTP Administration option in the IBM Navigator for i.

  1. If it’s not already started, start IBM Navigator for i by typing: STRTCPSVR *HTTP HTTPSVR(*ADMIN)
  2. Login to Navigator in your web browser on http://your-system:2001
  3. Sign-in with a user id/password that has authority to change the configuration
  4. Expand: IBM i Management (if not already expanded)
  5. Click “Internet Configurations”
  6. Click “IBM Web Administration for i”
  7. Sign in again (sigh – I hope IBM changes that so signing in twice isn’t needed)
  8. Click on the “HTTP Servers” tab
  9. Click “Create HTTP Server”
    • NOTE: do not use Create Web Services server, that is for IWS only
  10. Give your server a name and a description
    • The name can be any valid IBM I object name – I’ll use YAJLSERVER
  11. Click Next
  12. Accept the default server root by clicking “Next”
  13. Accept the default document root by clicking “Next”
  14. Keep the default of “All IP Addresses”, but change the port number to one that you’re not using for anything else
    • In my example, I will use 12345
  15. Click “Next”
  16. For the “access log”, I typically turn them off unless I want to audit the use of my server
    • To do this, click “no” and then “Next”
  17. When it asks how long to keep the logs (meaning the error logs, since we just turned off the access logs) I take the default of 7 days
  18. Click “Next”.
  19. It will then show a summary screen – Click “Finish”

You have now created a basic HTTP server using the IBM wizard, and it will be placed on the configuration page for your new server. Click “Edit Configuration File” (on the left, towards the bottom) to make changes to the configuration.

The basic HTTP server provides access to download HTML, images, etc. from the IFS. Since that won’t be needed in a web service, I recommend deleting the following section of configuration (simply highlight the section and press the delete key):

<Directory /www/yajlrest/htdocs>
   Require all granted
</Directory>

To enable access to the web services in your library, add the following configuration directives to the end of the file:

ScriptAliasMatch /rest/([a-z0-9]+)/.* /qsys.lib/yajlrest.lib/$1.pgm
<Directory /qsys.lib/yajlrest.lib>
   SetEnv QIBM_CGI_LIBRARY_LIST “YAJL;MYDATA;MYSRV;YAJLREST”
   Require all granted
</Directory>

The ScriptAliasMatch maps a URL to a program call. In this case, a URL beginning with /rest/ will call a program in the YAJLREST library. The part that is in parenthesis says that the program name must be made from the letters a-z or digits 0-9, and must be at least one character long. This is a regular expression and can be replaced with a different one if you have particular needs for the program name. In any case, the part of the URL that matches the part in parenthesis will be used to replace the $1 in the program name. This way, you can call any program in the YAJLREST library. The phrase “require all granted” allows everyone access to the YAJLREST library and is appropriate for an API that is available to the public.

The QIBM_CGI_LIBRARY_LIST environment variable controls the library list that your program will run with. In this case, it is adding the YAJL, MYDATA, MYSRV and YAJLREST libraries to the library list. MYDATA and MYSRV are meant as placeholders for your own libraries. YAJL is the default library that the YAJL tool will be installed into, and YAJLREST is where your program will be. Make sure you replace these with the appropriate libraries for your environment.

What if you don’t want your API to be available to the public? One way is to ask the IBM HTTP server to require a user id and password, as follows:

ScriptAliasMatch /rest/([a-z0-9]*)/.* /qsys.lib/yajlrest.lib/$1.pgm
<Directory /qsys.lib/yajlrest.lib>
   SetEnv QIBM_CGI_LIBRARY_LIST “YAJL;MYDATA;MYSRV;YAJLREST”
   require valid-user
   AuthType basic
   AuthName “REST APIs”
   PasswdFile %%SYSTEM%%
   UserId %%CLIENT%%
</Directory>

Some of these options are the same as the previous example. What’s different is that it no longer does “all granted” but instead requires a valid user. Which users are valid is based on the system’s password file (aka your user profiles). When your RPG program runs, it will use the user id supplied by the client (aka the user id and password that it checked against the user profiles.) If this is run in an interactive program, the phrase “REST APIs” will be printed to tell the user what he or she is signing on to.

If you use this sign-on option, I recommend also setting up SSL to prevent people’s passwords from being sent over the Internet in plain text.

There are many variations on the way users are authorized and who is allowed to access your setup. This is just one common example. To learn more about the different options, and to learn how to configure SSL, I recommend reading more in the IBM Knowledge Center. If you get really stuck, e-mail me at commonblog@scottklement.com.

Once your configuration is set up, click the “start” button to start the server. The start button is near the top of the screen, colored green and looks similar to the play button you’d find on a tape or CD player. You now have an HTTP server that will launch your RPG programs. The next step is to write the web service itself using RPG and YAJL.

The Basics of REST

When I design a REST web service (also known as REST API), I try to think of the URL as being a unique identifier for the business object that I’m working with. For example, when working with an invoice, the URL might represent a specific invoice by ending with the invoice number. Or, likewise, when working with customer information, the URL might represent a unique customer by ending with their customer number.

Once I’ve decided upon what the URL identifies, I think of what types of things I might want to do with the data. The REST paradigm assumes that you will use some (or all) of the following HTTP methods:

GET = Used to retrieve data identified by the URL

PUT = Used to set data identified by the URL in an idempotent way

POST = Used to set data identified by the URL in a non-idempotent way

DELETE = Remove the data identified by the URL

The term “idempotent” means that multiple calls will result in the same thing. For example, if I set X=1 that is idempotent. I can set X=1 once or 10 times; the result will still be 1. On the other hand, if I coded X=X+1 and ran it once, X would be 1, but if I ran it 10 times, it would be 10. Since multiple calls do not result in the same value, X=X+1 would be considered a non-idempotent statement.

Once I’ve thought about which of these HTTP methods should be supported by my service, I decide how the data will be passed to it. I do that by figuring out a JSON format for the data that is sent to my API as input, as well as another JSON document that’s returned with the output.

Let’s take a look at an example:

The Customer Details Example

To create a simple example of a REST API that’s easy to understand, I will use customer details as a simple example. The idea is that a URL will not only tell the HTTP server which program to call, but it will also identify a unique customer by its customer number. With that in mind, the URL will look like this:

http://our-system:12345/rest/custdetail/1500

Please note:

  • If using SSL, the “http:” would be replaced with “https:”
  • The 1500 at the end is an example of a customer number
  • When using features that don’t require an existing customer (such as adding a new one), the customer number should not be added
    • In that case the URL would be http://our-system:12345/rest/custdetail/
  • Since our URL contains “custdetail” after the /rest/ part of the URL, the ScriptAlias we configured will effectively do the same thing as CALL PGM(YAJLREST/CUSTDETAIL)

In my example, I want the API to be able to retrieve a list of customer information, retrieve the details of a specific customer, update a customer, create a new customer and delete old customers. For that reason, I will use the following HTTP methods:

  • GET = If the URL represents a unique customer, GET will return the details for that customer
    • If it does not reference a specific customer, it will instead return a full list of the customers available
  • PUT = Set customer information in an idempotent way
    • This will be used with an existing customer to set details such as its address – it cannot be used to add new customers since that would be non-idempotent.
  • POST = Add a new customer
    • This creates a new customer – since multiple calls to the URL would result in multiple customer records being created, this is non-idempotent.
  • DELETE = Remove customer details

One easy way to understand these is to think of them the same way you would think of database operations. GET is like SELECT/FETCH/READ, PUT is like UPDATE, POST is like INSERT/WRITE and DELETE is like DELETE. This isn’t a perfect analogy since it’s possible for an update database operation to be non-idempotent, but aside from that detail, they are very similar.

Since we’ve now determined what the URL represents, and what the methods will do, the other important idea is to determine the format of the data that will be sent or received.  For this example, I chose this format:

{
   “success”: true,
   “errorMsg”: “Error message goes here when success=false”,
   “data”: {
      “custno”: 496,
      “name”: “Acme Foods”,
      “address”: {
         “street”: “123 Main Street”,
         “city”: “Boca Raton”,
         “state”: “FL”,
         “postal”: “12345-6789”,
      }
   }
}

When a GET operation is done with a URL that identifies a unique customer, the above JSON document will be returned with the customer details (or an error message if there was an error.)

When using GET to list all customers, the same format will be used except that the “data” element will be converted into an array so that multiple customers can be returned at once.

When using the POST or PUT method, this document will be sent from the caller to represent the new values that the customer details should be set to. The POST and PUT operations will also return a document in the same format to show the caller what the customer details look like after the changes have been made.

When using the DELETE method, the row will be deleted. However, we will still use this document to return what the customer data was before the DELETE and also as a way to send errors, if any were found.

Coding the Example in RPG

Now that I’ve decided what the service will do, it’s time to code it!

The first thing my RPG program needs to know is whether GET, PUT, POST or DELETE was used. It can retrieve that by getting the REQUEST_METHOD environment variable. The IBM HTTP server will always set that variable to let us know which HTTP method was used. The RPG code to retrieve it looks like this:

   dcl-c UPPER const(‘ABCDEFGHIJKLMNOPQRSTUVWXYZ’);
   dcl-c lower const(‘abcdefghijklmnopqrstuvwxyz’);

   dcl-s env pointer;
   dcl-s method varchar(10);

   dcl-pr getenv pointer extproc(*dclcase);
      var pointer value options(*string);
   end-pr;

   .
   .

   env = getenv(‘REQUEST_METHOD’);
   if env <> *null;
      method = %xlate(lower: UPPER: %str(env));
   endif;

The getenv() routine returns a pointer to a C-style string, so I use the %STR built-in function to convert it to an RPG varchar variable. I’m also using the %XLATE built-in function to convert from lowercase to uppercase to ensure that the data will always be uppercase when I use it later in the code.

The next thing it will need is to get the customer number out of the URL (if any was supplied). The HTTP server provides the URL in an environment variable named REQUEST_URI that can be retrieved with the following code:

   dcl-s url varchar(1000);

   env = getenv(‘REQUEST_URI’);
   if env <> *null;
      url = %xlate(UPPER: lower: %str(env));
   endif;

The result is a variable named “url” that I’ve also converted to all uppercase. From that URL, I can retrieve the customer number by looking for it after the string “/rest/custdetail/”. Since I know “/rest/custdetail/” will always be in the URL (unless something went wrong), I can just scan for it and use that to get the position where the customer number is found. If there is no customer number, I’ll simply set my internal customer id to 0 and a code later in the program will use that to determine if a customer number was provided.

   dcl-c REQUIRED_PART const(‘/rest/custdetail/’);
   dcl-s pos int(10);
   dcl-s custpart varchar(50);
   dcl-s custid packed(5: 0);

   monitor;
      pos = %scan(REQUIRED_PART:url) + %len(REQUIRED_PART);
      custpart = %subst(url: pos);
      custid = %int(custpart);
   on-error;
      custid = 0;
   endmon;

Now that I know the method and customer number, I can choose what to do with it.  The code to handle GET is different than the code to handle PUT, for example, so I will select which section of code to run using a SELECT/WHEN group.

   select;
   when method = ‘GET’ and custid = 0;
      // code to list all customers
   when method = ‘GET’;
      // code to retrieve one customer
   when method = ‘PUT’;
      // code to update a customer (idempotent)
   when method = ‘POST’;
      // code to write a new customer (non-idempotent)
   when method = ‘DELETE’;
      // code to delete a customer
   endsl;

I will not provide all the code to do each of these options in the blog post, but instead at the bottom of this article because most of it is just standard RPG code that reads and writes data from the database. If you do want to see the program in its entirety, I will provide a link at the end of the article that you can use to download all the source code.

Working with JSON Using YAJL

The other part of the RPG code that I’d like to explain in this article is the code that works with JSON data. Although the DATA-INTO opcode may make this simpler in the future, as I write this, I haven’t had a chance yet to try it out. What I can show you is the “old fashioned” way of using YAJL to read and JSON data in an RPG program.

Although there are other ways to read JSON, I recommend using the “tree” style approach. I’ve found this to be the easiest, and in my RPG tools for YAJL, I provide a routine called yajl_stdin_load_tree that gets the data from the HTTP server and loads it into a YAJL tree in one subprocedure call.

   dcl-s docNode like(yajl_val);
   dcl-s errMsg varchar(500);

   docNode = yajl_stdin_load_tree(*on: errMsg);
   if errMsg <> ;
      // JSON was invalid, return an error message
   endif;

Now the JSON data has been loaded into a “tree-like” structure that is inside YAJL. The docNode variable is a pointer that points to the “document node”, which is a fancy way of saying that it points to the { and } characters (JSON object element) that surrounds the entire document. I can use the yajl_object_find procedure to locate a particular field inside that object and return a pointer to it. For example:

   dcl-s node like(yajl_val);

   node = yajl_object_find(docNode: ‘errorMsg’);
   if node = *null;
      cust.errorMsg =;
   else;
      cust.errorMsg = yajl_get_string(node);
   endif;

In this example, I retrieve a pointer to the value of the “errorMsg” field that is inside the JSON object. If “errorMsg” was not found, the pointer will be set to *NULL, in which case I know there wasn’t an error message sent. If it was sent, I can use the yajl_get_string subprocedure to get an RPG character string from the pointer.

In this case, the errorMsg field will be a character string, but other JSON fields might be numeric or boolean. I can retrieve these by calling other YAJL routines, such as yajl_get_number, yajl_is_true, and yajl_is_false in place of the yajl_get_string in the preceding example.

When a subfield has been set to an object or array, there are several different options to process it. I’ve already shown an example of yajl_object_find to process a single field inside of an object, but in addition to that, there are the following routines:

  • yajl_object_loop = loops through all of the fields in an object, one at a time
  • yajl_array_loop = loops through all of the elements in an array, one at a time
  • yajl_array_elem = get a particular array element by its index

Since this article is already very long, I will not provide examples of these in the post. Instead I will refer you to the link at the end of the article where you can download the complete example.

Once you’ve read your JSON data into your RPG program, you’ll want to ask YAJL to free up the memory that it was using. Remember, YAJL loaded the entire document into a tree structure, and that is taking up some storage on your system.  t’s very easy to free up the storage by calling yajl_tree_free and passing the original document node.

   yajl_tree_free(docNode);

YAJL will also be used to generate the JSON document that is sent back. For example, when you ask for the details of a customer, it uses YAJL to generate the response. The code in RPG looks like this:

   yajl_genOpen(*on);
   yajl_beginObj();

   yajl_addBool(‘success’: cust.success);
   yajl_addChar(‘errorMsg’: cust.errorMsg);

   if cust.success = *on;

      yajl_beginObj(‘data’);

      yajl_addNum(‘custno’: %char(cust.data.custno));
      yajl_addChar(‘name’: %trim(cust.data.name));

      yajl_beginObj(‘address’);
      yajl_addChar(‘street’: %trim(cust.data.address.street));
      yajl_addChar(‘city’:   %trim(cust.data.address.city));
      yajl_addChar(‘state’%trim(cust.data.address.state));
      yajl_addChar(‘postal’: %trim(cust.data.address.postal));
      yajl_endObj();

      yajl_endObj();

   endif;

   yajl_endObj();

   if cust.success;
      yajl_writeStdout(200: errMsg);
   else;
      yajl_writeStdout(500: errMsg);
   endif;

   yajl_genClose();

Prior to running this routine, the RPG program has loaded all the customer information into the “cust” data structure. The preceding routine uses that database data together with YAJL’s generator routines to create a JSON document.

The yajl_genOpen procedure starts the YAJL generator. It accepts one parameter, which is an indicator that determines whether the generated document is “pretty” or not. When the indicator is on, it will format the data with line feeds and tabs so that it’s easy for a human being to read. When the indicator is off, it will use the smallest number of bytes possible by removing line feeds tabs and extra spacing so that the entire JSON document is one big line of text. The “not pretty” version is a little more efficient for the computer to process, but since it is harder to read, I typically pass *ON while testing my program and change it to *OFF for production use.

When the JSON document calls for a new object (identified by the { and } characters), you can create it by calling yajl_beginObj. This will output the { character as well as make any subsequent fields be subfields of the new object. When you’re done creating the object, you can use yajl_endObj to output the } character and end the object.

There is a similar set of routines named yajl_beginArray and yajl_endArray that create a JSON array (vs. an object) that I did not use in this example but are useful when an array is called for.

Adding a field to an object or array is done by calling the yajl_addNum, yajl_addChar and yajl_addBool routines for numeric, character and boolean JSON fields, respectively. These routines not only add the fields, but they take care of escaping any special characters for you so that they conform to the JSON standard.

A really good way to understand the generator is to compare the expected JSON document (as I described in the section titled “The Customer Details Example” above) to the statements that generate it. You’ll see that each time a new object is needed, it calls yajl_beginObj, and each time a field is added to an object, it calls the appropriate “add” routine for the data type. These statements all correspond exactly to the JSON data that is output.

Once I’ve called the YAJL routines to add the data, I’ll have a JSON document, but it will be stored internally inside YAJL’s memory. To write it out to the HTTP server (which, in turn, will send it on to the program that is consuming our REST API), I use the yajl_writeStdout routine. This routine lets me provide an HTTP status code and an error message as a parameter. I recommend using a status of 200 to indicate that everything ran correctly or 500 to indicate that an error occurred.

Finally, once the JSON document has been sent, I call yajl_genClose to free up the memory that was used by the JSON document that YAJL generated.

Conclusion

YAJL is a powerful way to write JSON-based REST APIs. Although some of the examples may seem a little confusing at first, once you’ve successfully written your first one, you’ll get the hang of it quickly. Then it’ll be no problem to write additional ones.

You can learn more about YAJL and download the code samples that I mention in this post from my website at:

www.scottklement.com/yajl

If you’ll be at COMMON’s PowerUp18 conference, be sure to check out my session, Providing Web Services on IBM i, where I will demonstrate these techniques as well as other ways to provide your own web services in RPG.

Unlock Your RPG Applications to Enable Modern Online Experiences

By Dan Magid

What programming languages or frameworks do you think of when you hear web development? Probably JavaScript; maybe PHP or Python. But RPG? Not likely.

Nonetheless, for those who rely on longstanding IBM i systems to support their mission-critical operations, accessing current RPG applications is a necessity for powering fully functional, modern mobile applications. Of course, you might ask how a 60-year-old programming language could possibly handle the demands of modern online interactions.

RPG wasn’t originally built to power the web, but believe it or not, it may just be your fastest, most cost-effective option. With technological advances, it’s not always about what something was decades ago, but rather what it can do for you today.

In fact, RPG-based applications are being repurposed in ways their original developers could not have imagined, with amazing results. There is a significant amount of domain knowledge about the unique way your company does business built into your RPG applications. With the rise of the API economy, developers need that functionality for web, mobile and other applications. While RPG itself was not built with the web in mind, evolutionary technology has turned RPG into a web powerhouse, enabling the creation of web and mobile UIs—and even completely reengineered workflows—using the capabilities of existing RPG applications. And, RPG programs can call APIs from virtually any other application. Now, RPG code from 40 years ago underpins some of the most innovative applications you’ll see for call center and customer self-service applications, for example.

But why go this route when you could replace your old applications with pre-packaged or custom alternatives? The answer is that building upon what you already have—your venerable RPG applications— is faster, lower risk and significantly less expensive than replacing an entire system. Remember that those applications often have decades of business rules that must be replicated by any new system you buy or build from scratch, and that the TCO of your IBM i is on average about 1/3 of the most popular web and mobile platforms.* By keeping the back-end RPG application in place and using APIs or modern UIs, businesses can deliver renewed value from their current investments while still offering the kinds of features and experiences employees and customers need.

 

* Quark + Lepton, 2017 Study

Guest Blogger

Daniel Magid is Managing Director of Rocket Software’s Application Lifecycle Management & DevOps lab, and is a recognized authority on helping leading organizations achieve compliance through ALM solutions and DevOps best practices. He has written a variety of articles for leading IT publications and is a regular speaker at technology conferences.

The Strange World of Bug Bounty Hunters

How do software companies find dangerous bugs in their code? Ideally, their own QA departments discover them before it’s released. Sometimes they find out only when customers have problems. That might mean after there’s been a breach. But sometimes they hear about bugs from freelancers who find them in return for a reward. These people are called bug bounty hunters.

Some companies find it worthwhile to offer payment for bug reports. Learning about security holes before anyone can exploit them can save the company’s reputation, which is worth a lot of money. Recently Google paid $112,500 to a researcher for discovering a flaw that could have let a website push arbitrary code into an Android device. Having to deal with it after criminals found out could have been far more expensive.

The Mind of the Bounty Hunter

Bug hunting makes up half or more of some people’s income. They spend hours every day looking for flaws in websites. How different are they, really, from those who do the same thing and use their discoveries to steal information? Sometimes the same person plays both sides of the fence, depending on which one is paying better.

It’s the challenge, perhaps even more than the money, which motivates them. Anyone with those skills could get a well-paying job in QA. But they’d rather be on their own, chasing down bugs without reporting to a boss. Their attitude is, “So you think I can’t break this code? I’ll show you!”

The Benefit to Users

A bounty may encourage hackers to stay within the law. It can even motivate them to work harder at what they like to do. It’s easier to explain their income when it comes from Google rather than the Shadow Brokers, and there’s less chance of blackmail afterward. When they report bugs, the software publisher can fix them before anyone is harmed.

Software bounty hunters are a strange breed, there’s no question. But they do all of us some good.

Need to learn more about IT security? Take a look at POWERUp18 security sessions.

A Powerful Way to Run Unix and Open Source Tools from a Program

Perhaps the biggest area of growth in IBM i programming over the past several years has been the Open Source languages. There are thousands of utilities, mostly designed for Unix, that you can run in the QShell and PASE environments, and these have become very popular on IBM i! However, running these tools from your RPG and CL programs can be tricky. This post will introduce you to a free utility called UNIXCMD that makes it much easier.

Why is it Tricky?

It’s tricky because there’s a difference in the way IBM i and Unix systems run programs. When a program is called on Unix, a new “process” is created, this is very much like a new job on IBM i, except that it is created each time a program is called. This gives the calling program a choice, it can either stop and wait for the called program to finish, or it can continue and run simultaneously with the program it called. In a way, this is similar to submitting a batch job on i, but it’s different in that both programs can interact with the user’s display.

There are two common ways of running Unix programs, PASE via the QP2SHELL API and QShell via the STRQSH (or it’s alias QSH) CL command.

The QP2SHELL API runs a PASE program directly in the current job, which will cause problems if the job expects to be able to run simultaneously with the caller. Programs that use multiple threads, in particular, can have strange problems that are very hard to troubleshoot. The fix is to spawn a child job to run QP2SHELL so that it’s not in the same job as the caller. To enable input and output, you need to connect pipes to that child job. This is a lot of work, and often more than a programmer bargained for. (In fact, my description is somewhat oversimplified, to keep this brief!)

The STRQSH CL command solves this by always spawning its own child job and connecting pipes to it. It does all of this for you. The problem with this is that you are limited in how you can interact with the input and output streams. For a program to work with them, the only real option is to read and write from temporary files. This works, but it cumbersome, you have to create the file, clear it, redirect the I/O, and can only read the output once the whole process has finished.

Scott’s Solution

My solution is to create tools that do the work of submitting the child job for you, and connecting the pipes to a simple interface that is easy to use from your program. For a CL program, I’ve provided simple commands that open, read, write and close the connection, allowing you to read and write in a natural way. In RPG, I’ve used the Open Access interface so that you can open the connection with RPG’s native file interface, and read and write using the standard open, read, write and close opcodes.

The RPG Interface

Let’s take a look at RPG first. My initial examples use the newer “all-free” approach to writing RPG, but if you’re unable to use newer RPG, don’t fret – see the section titled “Older RPG Code”, below.
This example switches to the /QIBM directory in the IFS, lists the files in that directory, and prints them to the spool:

Notice the HANDLER keyword on the DCL-F statement. This tells RPG to access this file through the Open Access interface. When you run any of RPG’s file opcodes against this file, it will (under the covers) call routines in the UNIXCMDOA program, that allows my tool to take control and handle all of the work for you.

The command to run is provided in the second parameter to the HANDLER keyword. Since I’m setting that command in calculations in my program, I do not open the file until the command variable is set. For that reason, the file is declared with the USROPN keyword, and I open it explicitly with the OPEN opcode.

Unix utilities allow you to run multiple commands on a single line if you separate them with a semicolon. In this example, the cd command is used to switch to the /QIBM directory, and then the ls command is run afterwards to get the output.

To get the output from the Unix ls (list directory) command, I simply use the READ opcode. Since this is a program-described file (there are never any defined fields in a Unix input or output stream) I am using RPG’s feature that lets me read program-described data into a data structure.

When I’m done, I use the CLOSE opcode to shut down the Unix process. One thing that surprises people who are new to UNIXCMD is that any errors that occur in the background Unix program will be reported on the CLOSE opcode rather than the OPEN, READ or WRITE. This is because the Unix program has the opportunity to write error messages, and will not report that it has failed until the program has ended. Catching errors can be done easily with RPG’s MONITOR and ON-ERROR opcodes.

By default, the UNIXCMD utility runs your program using the QShell interface. If you prefer to run the PASE interface and avoid the QShell environment, you can do that by prefixing the command string with “pase:”, as shown in the next example.

The examples so far have only read output from a Unix command. In the next example, I’d like to demonstrate sending data both ways. In this case, I’m calling a PHP script that calls a web service to Geocode and address. In other words, I pass an address as input, and the script returns the latitude and longitude coordinates where that address can be found. To do that, I’ve written a PHP script that receives the address from it’s “standard input” (that is, the pipe that is connected to its input stream) and write the coordinates to standard output. (If you’re interested in the PHP code, it is included in the downloadable examples on my web site.) To call it from RPG, I can simply write my data to it, and then use the READ opcode to get the results.

The first thing you’ll notice in this statement is that it sets the Unix PATH variable. This is done because many people don’t have the PHP command in their PATH. PATH is very much like a library list, except that it is a list of IFS directories that the Unix environment uses to find a command. The PATH statement adds the directory where we’ve placed the php-cli command (CLI stands for “command line interface”) so that QShell can find it.

Another important thing to note is that data sent as input through the pipe is not automatically converted from EBCDIC to ASCII or Unicode. To solve that problem, I added a call to the QShell iconv utility, which can translate between different CCSIDs. In this case, will convert between 0 (a special value that means “this job’s CCSID”) and iso-8859-1 which is CCSID 819 and is a flavor of ASCII.

A Note About Input and Output

UNIXCMD assumes that you will write all the data that is sent as input to the Unix command first, before the first time you read its output. When you read the Unix output, it will shut down the Unix input stream to signal the Unix program that no more data is coming. This works well in most applications.

However, if you would prefer that it not close the stream, this can be made with a very simple code change to the UNIXCMD utility. If this would be useful to you, please e-mail me at commonblog@scottklement.com and I’ll be glad to show you how to change it.

Older RPG Code

Sadly, not everyone has a current version of RPG. To help those people, I’ve provided a way of using UNIXCMD that’s compatible with even the oldest versions of RPG IV using the SPECIAL file interface. Here is the last example rewritten to use that approach.

Notice that the command is passed to the SPECIAL file through the PLIST. The second parameter in the PLIST is not shown in this example, but you can add a 1 character second parameter and set it to P if you want to run in the PASE environment. If you do not pass this parameter (or set it to a Q) it will run QShell instead. Here is an excerpt of code that does that:

Since the SPECIAL file approach does not require Open Access, it will work all the way back to V5R3.

Using the UNIXCMD Tool from CL

Since the CL programming language does not support Open Access, there is no way to use the standard IBM supplied SNDF or RCVF opcodes that you are used to using for a normal file. Instead, I have created my own CL commands named OPNPIPE, SNDPIPE, RCVPIPE and CLOPIPE that handle the open, send, receive and close functions, respectively. The OPNPIPE command accepts the command to run, and lets you designate whether it is PASE or QShell. Aside from these differences, the UNIXCMD utility works the same from CL as it does from RPG.

Here’s an example of listing files (like the first RPG example) using the CL interface:

Get the UNIXCMD Utility

UNIXCMD is an open source tool that is available at no charge. You can download it, the examples given in this article, and a few more examples from my web site at the following link:

www.scottklement.com/unixcmd

The Python Programming Language

Python is a powerful, interpreted, open-source language available for most operating systems, including IBM i. It has many uses and has a very easy to learn syntax. Many packages are available to add capabilities not included in the Python core. Python is a great language to use for general purpose programming of all sorts.

Open Source - Python

 

 

 

 

Python Language Details

Python structure uses white space instead of brackets. This makes the code more readable than in similar languages such as Perl. The syntax is logical and easy to follow, even for beginners, and typing is dynamic. Even with this simple syntax, the language also has great flexibility and powerful capabilities. Object-oriented programming is possible but not required, as it is in other languages like Java or Ruby.

Web Development

One use of Python is for back-end web development. Frameworks such as Flask and Django simplify back-end development using built-in components that are popular. Rapid development and prototyping are possible using these frameworks. Python is also good for general scripting, including projects involving crawling the web or scraping information from websites.

Data Science

Python is a great language to use for data science. Useful libraries include NumPy, which adds speedy mathematical capabilities, and SciPy, which adds the ability to manipulate complex data in arrays for linear algebra and other purposes. Several other libraries are useful for doing data science in Python.

Philosophy

Python has a philosophy often referred to as “The Zen of Python“. This philosophy encourages simple, clean code that is easy to read and explain to others. It has several other suggestions for optimizing the use of Python to make it uniform and ideal using best practices.

Summary

A simple yet powerful open-source language, Python is a great choice for many projects. Using Python gives you many capabilities out of the box and additional packages to add more. The language is quite readable, and its uses include many technical domains.

Learn Python with COMMON’s Open Source Video Series.

 

Python