Stop Limiting Yourself to 10 Character Field Names in RPG Files

At POWERUp18 this May, someone asked me “When will RPG’s native file access support long names from databases and display files?” The answer is: it already does and has for quite some time! Originally it required reading into data structures, but that’s not even true anymore. Today, RPG’s native file access fully supports long names!

Since having that conversation, I’ve been asking around. It seems that not many RPGers know about this very useful feature. So in today’s post, I’ll show you how it works.

What I Found Out When Asking Around

I didn’t do any sort of official survey, and these statistics weren’t achieved scientifically, but…while I was informally asking around, here’s what I discovered:

  • “Everyone” knows you can create column (field) names longer than 10 characters using SQL
  • “Everyone” knows you can read, insert, update and delete using long field names in SQL
  • About half of the people know that DDS also supports long field names
  • About 25% thought RPG could use long names if you use data structures for I/O
  • Only a few people (maybe 5%) knew RPG supports long names without data structures

We can do it with or without data structures! The data structure support has been around for a long time, and perhaps that’s why more people are familiar with it. (But, it can be awkward to code.)

The ability to use long names without data structures was added in 2014. If you have the latest PTFs for RPG on IBM i 7.1 or 7.2, then you have this support already. It was also included in the RPG compiler that was shipped with IBM i 7.3 in 2016. If you’re running 7.3, PTFs aren’t even needed.

Defining Tables with Long Names in SQL

Defining a field name longer than 10 characters is very natural in SQL. You simply define your table with a statement like this:

What may not be obvious, however, is that when a field is longer than 10 characters, there are actually two names assigned to it. The regular “column name” and what SQL refers to as the “system name”, which is limited to 10 characters for compatibility with older tools like CPYF, OPM languages or CL programs. In the above example, the system will generate names like CUSTO00001 for CUSTOMER_NUMBER and COMPA00001 for COMPANY_NAME.

Since there are still a lot of places where you might want to use the shorter system names, I recommend explicitly naming them. That way your short names can be meaningful. For example:

When you are working on modernizing your databases, this dual-name approach is really handy because it lets you keep the old, short (and often ugly) names that you’ve always used, keeping the files compatible with older programs. New programs can use the longer names and be more readable. Eventually, you can rewrite all the programs to use the long names, so this is a great tool for modernizing!

Long Names in DDS (Including Displays and Printers!)

DDS also supports long names using the ALIAS keyword. The name “alias” makes sense if you think about it, since any time you have a field longer than 10 characters, there are two names and either can be used. In other words, they are aliases for each other.

The DDS equivalent would be coded like this:

Mind you, I don’t recommend defining your new physical files using DDS, but for existing files, it may be easier to simply add the ALIAS keyword than to re-do the file using SQL.

Another reason why it’s important to know about the ALIAS keyword in DDS is because it also supports externally defined display, printer and even ICF files. Since all of these types of files support long names and have a great mechanism for backward compatibility with programs that must use short names, there’s no reason not to take advantage of this!

Long Names in RPG Programs

Like DDS, RPG uses the term “alias” when referring to long names. Starting with IBM i 7.1, RPG was able to use this long name support in qualified external data structures. This original support also required you to qualify the record format names (by using the QUALIFIED keyword on the F-spec) because without this keyword, RPG generates input specs. At the time 7.1 was released, I-specs did not support long names.

This support that shipped with 7.1 could be somewhat awkward. For example, the code to update the last paid date for customer 1000 would look like this:

I found this awkward because you had to use separate data structures for input and output (though, a PTF was released later that relaxed that rule) and you had to qualify both the field and record format names. I don’t know if everyone agrees, but I found that cumbersome!

The 2014 update added full support for long names, without any need to use data structures at all! So, if you’re up-to-date on PTFs and are running IBM I 7.1 or newer (which you really should be), you can do this instead:

Naturally, the alias keyword works in free format code as well. Here’s an example:

Don’t forget that alias support extends to display files and printer files, too!  There’s no reason to use short cryptic field names anymore. Take advantage of the long ones. You’ll love it!

Parsing JSON with DATA-INTO! (And, Vote for Me!)

Finally, RPG can parse JSON as easily as it can XML! This month, I’ll take a look at RPG’s brand-new DATA-INTO opcode and how you can use it today in your RPG programs. I’ve combined DATA-INTO with the free YAJL tool to let you parse JSON documents with DATA-INTO in a manner very similar to the way you’ve been using XML-INTO for XML documents.

I’ll also take a moment to ask you to vote in COMMON’s Board of Directors election. I’m running for the Board this year, alongside some other great leaders in our community. Please spread the word and vote!

Board of Directors Election

I need your vote! I am running for COMMON’s Board of Directors, and this is an elected position voted upon by the COMMON membership. Why am I running? I’ve had a love affair with the IBM i community for a long time, and since 2001, I have tried very hard to bring value through articles, free software, online help and giving talks on IBM i subjects. To me, COMMON is one of the main organizations that represents the community, and I feel that this is one more way that I can provide value and give back.

Voting is open from April 22nd to May 22nd. You can learn more about my position as well as the other candidates by clicking here.

To place your vote, you’ll need to log into Cosmo and click the “Board of Directors Election” at the top of the page.

Please vote and help me spread the word! It means a lot!

RPG’s DATA-INTO Opcode: The Concept

Years ago, IBM added the XML-INTO opcode to RPG, which greatly simplified reading XML documents by automatically loading them into a matching RPG variable. Unfortunately, it only works with XML documents, and today, JSON has become even more popular than XML. This led me and many other RPGers to ask IBM for a corresponding JSON-INTO opcode. Instead, IBM had a better idea: DATA-INTO. DATA-INTO is an opcode that can potentially read any data format (with a little help) and place it into an RPG variable. That way, if the industry decides to change to another format in the future, RPG already has it covered.

I expect that you’re thinking what I was thinking: “There are millions of data formats! No single opcode could possibly read them all!” DATA-INTO solves that problem by working together with a third-party routine called a “parser”. The parser is what reads and understands data formats such as JSON or whatever might replace it in the future. It then passes that data back to DATA-INTO as parameters, and DATA-INTO maps the data into RPG variables. Think of the parser is a plug-in for DATA-INTO. As long as you have the right plug-in (or “parser”), you can read a data format. In this article, I’ll present a parser for JSON, so you’ll be able to read JSON documents. Other people have already published parsers for property file format and CSV documents. When “the next big thing” happens, you’ll only need a parser for it, and then DATA-INTO will be able to handle it.

One more way to think of it: it is similar to Open Access. Open Access provides a way to use RPG’s native file support to read any sort of data. Where traditional file access called IBM database routines under the covers, Open Access lets you provide your own routine, so you can read any sort of data and aren’t limited to only IBM’s database. DATA-INTO works the same way, except instead of replacing the logic for files, it replaces the logic for XML-INTO with a general-purpose system.

DATA-INTO Requirements

DATA-INTO is part of the ILE RPG compiler and runtime. No additional products need to be installed besides the aforementioned parser. You will need IBM i 7.2 or newer, and you’ll need the latest PTFs installed.

The PTFs required for 7.2 and 7.3 are found here.

When IBM releases future versions of the operating system (newer than 7.3), the RPG compiler will automatically include DATA-INTO support without additional PTFs.

DATA-INTO will not work with the 7.1 or earlier RPG compilers and will not work with TGTRLS(V7R1M0) or earlier specified, even if you are running it on 7.2.

My JSON parser for DATA-INTO requires the open source YAJL software. You will need a copy from April 17, 2018 (or newer) to have DATA-INTO support. It is available at no charge from my website.

DATA-INTO Syntax

DATA-INTO works almost the same as the XML-INTO opcode that you may already be familiar with. The syntax is as follows:

DATA-INTO your-result-variable %DATA(document : rpg-options) %PARSER(parser : parser-options);

-or-

DATA-INTO %HANDLER(handler : commArea) %DATA(document : rpg-options) %PARSER(parser : parser-options);

If you’re familiar with XML-INTO, you’ll notice that the syntax is identical except for the %PARSER built-in function and its parameters. Here’s a description of what each part means:

your-result-variable = An RPG variable to receive the data. Most often, this will be a data structure that is formatted the same way as the document you are reading. Fields in the document will be mapped to corresponding fields in your variable, based on the variable names. I’ll explain this in more detail below.

%DATA(document : rpg-options) = Specifies the document to be read and the options that are understood and used by RPG (as opposed to the parser) when mapping fields into your variable. The document parameter is either a character string containing the document itself or is an IFS path to where the document can be read from disk. The rpg-options parameter is a character string containing configuration options of how the data should be mapped into variables. (It is the same as the second parameter to the %XML built-in function used with XML-INTO and works the same way.) The following is a summary of those options:

  • path option = specifies a location within the JSON document to begin parsing, and lets you parse only a subset of the document if desired
  • doc option = specify doc=string if the document parameter contains the JSON document itself, or doc=file if it contains an IFS path name
  • ccsid option = controls whether RPG does it’s processing in Unicode or EBCDIC
  • case option = controls how strictly a variable name must match the document field names, including whether it’s case sensitive or whether special characters get converted to underscores
  • trim option = controls whether blanks and other whitespace characters are trimmed from character strings
  • allow missing option = controls whether it is okay for the document to be missing fields that are in the RPG variable
  • allow extra option = controls whether it is okay for the document to have extra fields that are not in the RPG variable
  • count prefix option = a prefix to be added for RPG fields that should contain a count of the number of matching elements (vs the data of the element)
    • For example, the number of entries loaded into an array
    • Can also be used to determine whether an optional element was/wasn’t provided

I don’t want this post to get too bogged down with the details of each option, so if you’d like to read more details, please see the ILE RPG reference manual.

%PARSER(parser : parser-options) = Specifies a program or subprocedure to act as a parser (or “plugin” as I like to think of it) that will interpret the document. My parser will be a JSON parser, and it will know how to interpret a JSON document and pass its data to RPG. The parser-options parameter is a literal or variable string that’s intended to be used to configure the parser. The format of parser-options, and which options are available, is determined by the code in the parser routine.

%HANDLER(handler : commArea) = Specifies a handler routine to be used as an alternative to a variable. You use this when your data should be processed in “chunks” rather than reading the entire document at once. I consider this to be a more advanced usage of DATA-INTO (or XML-INTO) that is primarily used when it’s not possible to fit all the needed data into a variable. This was very common back in V5R4 when variables were limited to 64 KB but is not so common today. For that reason, I will not cover it in this article. If you’re interested in learning more, you can read about it in the ILE RPG Reference manual, or e-mail me to suggest this as a topic for a future blog post.

The YAJLINTO Parser

The current version of YAJL ships with a program named YAJLINTO, which is a parser that DATA-INTO can use to read JSON documents. You never need to call YAJLINTO directly. Instead you use it with the %PARSER function. Let’s look at an example!

In a production application, you’d get a JSON document from parameter, API or file. To keep this example simple, I’ve hard-coded in my RPG program by assigning it to a character string as follows:

dcl-s json varchar(5000);

json = ‘{ +
     “success”: true, +
     “errorMsg”: “No error reported”, +
     “customers”:[{ +
       “name”: “ACME, Inc.”, +
       “address”: { +
        “street”: “123 Main Street”, +
        “city”: “Anytown”, +
        “state”: “WI”, +
        “postal”: “53129” +
       } +
     }, +
     { +
       “name”: “Industrial Supply Limited.”, +
       “address”: { +
        “street”: “1000 Green Lawn Drive”, +
        “city”: “Fancyville”, +
        “state”: “CA”, +
        “postal”: “95811” +
       } +
     }] +
    }’;

In last month’s blog entry, I explained quite a bit about JSON and how to process it with YAJL. I won’t repeat all of the details about how it works, but just to refresh your memory, the curly braces (the { and } characters) start and end a JSON data structure. (They call them “objects”, but it is the same thing as a data structure in RPG.) The square brackets (the [ and ] characters) start and end an array.

Like XML-INTO, DATA-INTO requires that a variable is defined that exactly matches the layout of the JSON document. When RPGers write me complaining of problems with this sort of programming, the problem is almost always that their variable doesn’t quite match the document. So please take care to make them match exactly. In this case, the RPG definition should look like this:

dcl-ds myData qualified;
success ind;
errorMsg varchar(500);
num_customers int(10);
dcl-ds customers dim(50);
name varchar(30);
dcl-ds address;
street varchar(30);
city  varchar(20);
state char(2);
postal varchar(10);
end-ds;
end-ds;
end-ds;

Take a moment to compare the RPG data structure to the JSON one. You’ll see that the JSON document starts and ends with curly braces and is therefore a data structure – as is the RPG. Since the JSON structure has no name, it does not matter what I name my RPG structure. So, I called it “myData”. The JSON document contains three fields named success, errorMsg and customers. The RPG code must also use these names so that they match. The customers field in JSON is an array of data structures, as is the RPG version. The address field inside that array of data structures is also a data structure, and therefore the RPG version must also be.

The one field that is different is “num_customers”. To understand that, let’s take a look at how I’m using these definitions with the DATA-INTO opcode.

data-into myData %DATA(json: ‘case=any countprefix=num_’)
         %PARSER(‘YAJLINTO’);

The first parameter to DATA-INTO is the RPG variable to read the data into, in this case it is the myData data structure shown above.

The JSON data is specified using the %DATA built-in function, which receives the ‘json’ variable – the character string containing the JSON data. The second parameter to %DATA is the options for RPG to use when mapping the fields. I did not need to specify “doc=string” to get the JSON data from a variable because “doc=string” happens to be the default. I did specify two other options, however.

case=any – means that the upper/lowercase of the RPG fields do not have to match that of the JSON fields.

countprefix=num_ – means that if I code a variable starting with the prefix “num_” in my data structure, it should not come from the data, but instead, RPG should populate it with the count of elements. In this case, since I have defined “num_customers” in my data structure, RPG will count the number of elements in the “customers” array and place that count into the “num_customers” field.

That explains why the extra num_customers field is in the RPG data structure. Since customers is an array and I don’t know how many I’ll be sent, RPG can count it for me, and I can use this field to see how many I received. That’s very helpful!

Countprefix is also helpful in cases where a field may sometimes be in the JSON document and sometimes may be omitted. In that case, the count prefixed field will bypass the “allow_missing” check and allow the field to not exist without error. If the field didn’t exist in the JSON, the count will be set to zero, and my RPG code can detect it and handle it appropriately.

The %PARSER function tells DATA-INTO which parser program to call. In this case, it is YAJLINTO. The %PARSER function is capable of specifying either a program or a subprocedure and can also include a library if needed.

When specifying a program as the parser, it can be in any of the following formats:

‘MYPGM’
‘*LIBL/MYPGM’
‘MYLIB/MYPGM’

Notice that the program and library names are in capital letters. Unless you are using case-sensitive object names on your IBM i (which is very unusual), you should always put the program name in capital letters.

To specify a subprocedure in a service program, use one of the following formats:

‘MYSRVPGM(mySubprocedure)’
‘MYLIB/MYSRVPGM(mySubprocedure)’
‘*LIBL/MYSRVPGM(mySubprocedure)’

The subprocedure name must be in parenthesis to denote that it is a subprocedure rather than a program name. Like the program names above, the service program name should be in all capital letters. However, subprocedure names in ILE are case-sensitive, so you will need to be sure to match the case exactly as it is exported from the service program. Use the DSPSRVPGM command to see the how the procedures are exported.

After the DATA-INTO opcode runs successfully, the myData variable will be filled in, and I can use its data in my program just as I would use any other RPG variable. For example, if I wanted to loop through the customers and display their names and cities, I could do this:

dcl-sint(10);

for x = 1 to myData.num_customers;
  dsply myData.customers(x).name;
  dsply myData.customers(x).address.city;
endfor;

Naturally, you wouldn’t want to use the DSPLY opcode in a production program, but it’s a really easy way to try it and see that you can read the myData data structure and its subfields. Now that you have data in a normal RPG variable, you can proceed to use it in your business logic. Write it to a file if you wish, or print it, place it on a screen, whatever makes sense for your application.

Parser Options for YAJLINTO

Earlier I mentioned that %PARSER has a second parameter for “parser options.” This parameter is optional, and I didn’t use it in the above example. However, there are some options available with YAJLINTO that you might find helpful.

Unlike the standard DATA-INTO (or XML-INTO) options, YAJLINTO expects its options to be JSON formatted. I designed it this way because YAJL already understands JSON format, so it was very easy to code. But, it is also powerful. I can add new features in the future (without interfering with the existing ones) simply by adding new fields to the JSON document.

As I write this, there are three options. All of them are optional, and if not specified, the default value is used.

value_true = the value that will be placed in an RPG field when the JSON document specifies a Boolean value that is true. By default, this puts “1” in the field because it’s assumed that Booleans will be mapped to RPG indicators. You can set this option to any alternate value you’d like to map, up to 100 characters long.

value_false = value placed in an RPG field when JSON document specifies a Boolean value that is false. The default value is “0”.

value_null = value placed in an RPG variable when a JSON document specifies that a field is null. Unfortunately, DATA-INTO does not have the ability to set an RPG field’s null indicator, so a special value must be placed in the field instead. The default is “*NULL”.

For example, consider the following JSON document:

json = ‘{ “inspected”: true, “problems_found”: false, +
     “date_shipped”: null }’;

In this example, I prefer “yes” and “no” for the Boolean fields. It’ll say, “yes it was inspected” or “no problems were found”. The date shipped is a date-type field and therefore cannot be set to the default null value of *NULL. So, I want to map the special value of 0001-01-01 to my date. I can do that as follows:

dcl-ds status qualified;
inspected varchar(3);
problems_found varchar(3);
date_shipped date;
end-ds;

data-into status %data(json:‘case=any’)
%parser(‘YAJLINTO’ : ‘{ +
“value_true”: “yes”,+
“value_false”: “no”,+
“value_null”: “0001-01-01” +
}’);

When this code completes, the inspected field in the status data structure will be set to “yes”, and the problems_found field set to “no”. The date_shipped will be set to Jan 1, 0001 (0001-01-01.)

Debugging and Diagnostics

and the parser during their processing. To enable this, you’ll need to add an environment variable to the same job as the program that uses data-into. For example, you could type this:

ADDENVVAR ENVVAR(QIBM_RPG_DATA_INTO_TRACE_PARSER) VALUE(‘*STDOUT’)

In an interactive job, this will cause the trace information to scroll up the screen as data-into runs. In a batch job, it would be sent to the spool. Information will be printed about what character sets were used and which fields and values were found in the document.

For example, in the case of the “status” data structure example in the previous section, the trace output would look like this:

—————- Start —————
Data length 136 bytes
Data CCSID 13488
Converting data to UTF-8
Allocating YAJL stream parser
Parsing JSON data (yajl_parse)
No name provided for struct, assuming name of RPG variable
StartStruct
ReportName: ‘inspected’
ReportValue: ‘yes’
ReportName: ‘problems_found’
ReportValue: ‘no’
ReportName: ‘date_shipped’
ReportValue: ‘0001-01-01’
EndStruct
YAJL parser status 0 after 68 bytes
YAJL parser final status: 0
—————- Finish ————–

Writing Your Own Parser

When DATA-INTO is run, it loads the document into memory and then calls the parser. It passes a parameter that contains the document to read as well as information about all of the options the user specified. It is then responsible for interpreting the document and calling some subprocedures to tell DATA-INTO what was found.

Writing a parser is best done by someone who is good at systems-type coding. You will need to work with pointers, procedure pointers, CCSID conversions and system APIs. I suspect most RPGers will want to find (or maybe purchase) a third-party parser rather than write their own. For that reason, I will not teach you how to write a parser in this blog entry.

However, if you’d like to learn more about this in a future installment of Scott’s iLand, please leave a comment below. If enough people are interested, I’ll write one.

In the meantime, I suggest the following options:

The Rational Open Access: RPG Edition manual has a chapter on how to write a parser and provides an example of a properties document parser.

Jon Paris and Susan Gantner recently published some articles that explain how to use DATA-INTO as well as write a parser. They provide an example of reading a CSV file.

Attending POWERUp18? Be sure to check out Scott’s sessions.

Interested in learning more about DATA-INTO? Attend this session from Barbara Morris.

See you in San Antonio!

RPG and YAJL, as Well as DATA-INTO News

This month, I’ll give you a quick update about RPG’s new DATA-INTO opcode and then focus on the basics of how you can host your own custom REST web service API written in RPG with the help of the standard IBM HTTP Server (powered by Apache) and the YAJL JSON library.

DATA-INTO News

IBM has posted documentation online for the new DATA-INTO opcode, and they promise that PTFs will be available on March 19th, 2018. That means that by the time this blog entry goes live, it’ll be available for you to install and try.

Unfortunately, I’m writing this before the PTFs are available, so I haven’t been able to try it yet. I hope to do so in time for next month’s blog.

Providing a Web Service with RPG and YAJL

I’ve been creating quite a few JSON-based REST APIs using the YAJL toolkit lately, and with DATA-INTO on the horizon, I thought it’d be a good time to give you an example. If you’re not familiar with YAJL, it is a free tool for reading and writing JSON documents. Since YAJL is written in C, I was able to compile it as a native ILE C service program and create an RPG front-end for it that makes it both easy to use and extremely efficient.

Of course, IBM provides us with the wonderful Integrated Web Services (IWS) tool that can do both REST and SOAP APIs using both JSON and XML. The advantage that IWS has over YAJL is that IWS does most of the work for you, simplifying things tremendously. On the other hand, YAJL is faster and lets you deal with much more complex documents. For the work that I do, I usually find YAJL works better.

Prerequisites

Before you can use the techniques that I describe in this post, you’ll need to have the following items installed:

  • IBM HTTP Server (powered by Apache)
    • This is available on the OS installation media as licensed program option 57xx-DG1 – you’ll want to see the IBM Knowledge Center for documentation about how to install it and its prerequisites
  • YAJL (Yet Another JSON Library)
  • ILE RPG compiler at version 6.1 or newer

Initial Setup

I recommend that you create a new library in which to put your programs that implement REST APIs. It’s certainly possible to put them in an existing library, but using a separate one helps with security. If you set up the HTTP server to only access the new library, you won’t need to worry about someone trying to use your server to run programs that they shouldn’t.

In my example, I will create a library named YAJLREST.

CRTLIB LIB(YAJLREST)

You’ll also need to set up the IBM HTTP server to point to your new library. This is best done using the HTTP Administration option in the IBM Navigator for i.

  1. If it’s not already started, start IBM Navigator for i by typing: STRTCPSVR *HTTP HTTPSVR(*ADMIN)
  2. Login to Navigator in your web browser on http://your-system:2001
  3. Sign-in with a user id/password that has authority to change the configuration
  4. Expand: IBM i Management (if not already expanded)
  5. Click “Internet Configurations”
  6. Click “IBM Web Administration for i”
  7. Sign in again (sigh – I hope IBM changes that so signing in twice isn’t needed)
  8. Click on the “HTTP Servers” tab
  9. Click “Create HTTP Server”
    • NOTE: do not use Create Web Services server, that is for IWS only
  10. Give your server a name and a description
    • The name can be any valid IBM I object name – I’ll use YAJLSERVER
  11. Click Next
  12. Accept the default server root by clicking “Next”
  13. Accept the default document root by clicking “Next”
  14. Keep the default of “All IP Addresses”, but change the port number to one that you’re not using for anything else
    • In my example, I will use 12345
  15. Click “Next”
  16. For the “access log”, I typically turn them off unless I want to audit the use of my server
    • To do this, click “no” and then “Next”
  17. When it asks how long to keep the logs (meaning the error logs, since we just turned off the access logs) I take the default of 7 days
  18. Click “Next”.
  19. It will then show a summary screen – Click “Finish”

You have now created a basic HTTP server using the IBM wizard, and it will be placed on the configuration page for your new server. Click “Edit Configuration File” (on the left, towards the bottom) to make changes to the configuration.

The basic HTTP server provides access to download HTML, images, etc. from the IFS. Since that won’t be needed in a web service, I recommend deleting the following section of configuration (simply highlight the section and press the delete key):

<Directory /www/yajlrest/htdocs>
   Require all granted
</Directory>

To enable access to the web services in your library, add the following configuration directives to the end of the file:

ScriptAliasMatch /rest/([a-z0-9]+)/.* /qsys.lib/yajlrest.lib/$1.pgm
<Directory /qsys.lib/yajlrest.lib>
   SetEnv QIBM_CGI_LIBRARY_LIST “YAJL;MYDATA;MYSRV;YAJLREST”
   Require all granted
</Directory>

The ScriptAliasMatch maps a URL to a program call. In this case, a URL beginning with /rest/ will call a program in the YAJLREST library. The part that is in parenthesis says that the program name must be made from the letters a-z or digits 0-9, and must be at least one character long. This is a regular expression and can be replaced with a different one if you have particular needs for the program name. In any case, the part of the URL that matches the part in parenthesis will be used to replace the $1 in the program name. This way, you can call any program in the YAJLREST library. The phrase “require all granted” allows everyone access to the YAJLREST library and is appropriate for an API that is available to the public.

The QIBM_CGI_LIBRARY_LIST environment variable controls the library list that your program will run with. In this case, it is adding the YAJL, MYDATA, MYSRV and YAJLREST libraries to the library list. MYDATA and MYSRV are meant as placeholders for your own libraries. YAJL is the default library that the YAJL tool will be installed into, and YAJLREST is where your program will be. Make sure you replace these with the appropriate libraries for your environment.

What if you don’t want your API to be available to the public? One way is to ask the IBM HTTP server to require a user id and password, as follows:

ScriptAliasMatch /rest/([a-z0-9]*)/.* /qsys.lib/yajlrest.lib/$1.pgm
<Directory /qsys.lib/yajlrest.lib>
   SetEnv QIBM_CGI_LIBRARY_LIST “YAJL;MYDATA;MYSRV;YAJLREST”
   require valid-user
   AuthType basic
   AuthName “REST APIs”
   PasswdFile %%SYSTEM%%
   UserId %%CLIENT%%
</Directory>

Some of these options are the same as the previous example. What’s different is that it no longer does “all granted” but instead requires a valid user. Which users are valid is based on the system’s password file (aka your user profiles). When your RPG program runs, it will use the user id supplied by the client (aka the user id and password that it checked against the user profiles.) If this is run in an interactive program, the phrase “REST APIs” will be printed to tell the user what he or she is signing on to.

If you use this sign-on option, I recommend also setting up SSL to prevent people’s passwords from being sent over the Internet in plain text.

There are many variations on the way users are authorized and who is allowed to access your setup. This is just one common example. To learn more about the different options, and to learn how to configure SSL, I recommend reading more in the IBM Knowledge Center. If you get really stuck, e-mail me at commonblog@scottklement.com.

Once your configuration is set up, click the “start” button to start the server. The start button is near the top of the screen, colored green and looks similar to the play button you’d find on a tape or CD player. You now have an HTTP server that will launch your RPG programs. The next step is to write the web service itself using RPG and YAJL.

The Basics of REST

When I design a REST web service (also known as REST API), I try to think of the URL as being a unique identifier for the business object that I’m working with. For example, when working with an invoice, the URL might represent a specific invoice by ending with the invoice number. Or, likewise, when working with customer information, the URL might represent a unique customer by ending with their customer number.

Once I’ve decided upon what the URL identifies, I think of what types of things I might want to do with the data. The REST paradigm assumes that you will use some (or all) of the following HTTP methods:

GET = Used to retrieve data identified by the URL

PUT = Used to set data identified by the URL in an idempotent way

POST = Used to set data identified by the URL in a non-idempotent way

DELETE = Remove the data identified by the URL

The term “idempotent” means that multiple calls will result in the same thing. For example, if I set X=1 that is idempotent. I can set X=1 once or 10 times; the result will still be 1. On the other hand, if I coded X=X+1 and ran it once, X would be 1, but if I ran it 10 times, it would be 10. Since multiple calls do not result in the same value, X=X+1 would be considered a non-idempotent statement.

Once I’ve thought about which of these HTTP methods should be supported by my service, I decide how the data will be passed to it. I do that by figuring out a JSON format for the data that is sent to my API as input, as well as another JSON document that’s returned with the output.

Let’s take a look at an example:

The Customer Details Example

To create a simple example of a REST API that’s easy to understand, I will use customer details as a simple example. The idea is that a URL will not only tell the HTTP server which program to call, but it will also identify a unique customer by its customer number. With that in mind, the URL will look like this:

http://our-system:12345/rest/custdetail/1500

Please note:

  • If using SSL, the “http:” would be replaced with “https:”
  • The 1500 at the end is an example of a customer number
  • When using features that don’t require an existing customer (such as adding a new one), the customer number should not be added
    • In that case the URL would be http://our-system:12345/rest/custdetail/
  • Since our URL contains “custdetail” after the /rest/ part of the URL, the ScriptAlias we configured will effectively do the same thing as CALL PGM(YAJLREST/CUSTDETAIL)

In my example, I want the API to be able to retrieve a list of customer information, retrieve the details of a specific customer, update a customer, create a new customer and delete old customers. For that reason, I will use the following HTTP methods:

  • GET = If the URL represents a unique customer, GET will return the details for that customer
    • If it does not reference a specific customer, it will instead return a full list of the customers available
  • PUT = Set customer information in an idempotent way
    • This will be used with an existing customer to set details such as its address – it cannot be used to add new customers since that would be non-idempotent.
  • POST = Add a new customer
    • This creates a new customer – since multiple calls to the URL would result in multiple customer records being created, this is non-idempotent.
  • DELETE = Remove customer details

One easy way to understand these is to think of them the same way you would think of database operations. GET is like SELECT/FETCH/READ, PUT is like UPDATE, POST is like INSERT/WRITE and DELETE is like DELETE. This isn’t a perfect analogy since it’s possible for an update database operation to be non-idempotent, but aside from that detail, they are very similar.

Since we’ve now determined what the URL represents, and what the methods will do, the other important idea is to determine the format of the data that will be sent or received.  For this example, I chose this format:

{
   “success”: true,
   “errorMsg”: “Error message goes here when success=false”,
   “data”: {
      “custno”: 496,
      “name”: “Acme Foods”,
      “address”: {
         “street”: “123 Main Street”,
         “city”: “Boca Raton”,
         “state”: “FL”,
         “postal”: “12345-6789”,
      }
   }
}

When a GET operation is done with a URL that identifies a unique customer, the above JSON document will be returned with the customer details (or an error message if there was an error.)

When using GET to list all customers, the same format will be used except that the “data” element will be converted into an array so that multiple customers can be returned at once.

When using the POST or PUT method, this document will be sent from the caller to represent the new values that the customer details should be set to. The POST and PUT operations will also return a document in the same format to show the caller what the customer details look like after the changes have been made.

When using the DELETE method, the row will be deleted. However, we will still use this document to return what the customer data was before the DELETE and also as a way to send errors, if any were found.

Coding the Example in RPG

Now that I’ve decided what the service will do, it’s time to code it!

The first thing my RPG program needs to know is whether GET, PUT, POST or DELETE was used. It can retrieve that by getting the REQUEST_METHOD environment variable. The IBM HTTP server will always set that variable to let us know which HTTP method was used. The RPG code to retrieve it looks like this:

   dcl-c UPPER const(‘ABCDEFGHIJKLMNOPQRSTUVWXYZ’);
   dcl-c lower const(‘abcdefghijklmnopqrstuvwxyz’);

   dcl-s env pointer;
   dcl-s method varchar(10);

   dcl-pr getenv pointer extproc(*dclcase);
      var pointer value options(*string);
   end-pr;

   .
   .

   env = getenv(‘REQUEST_METHOD’);
   if env <> *null;
      method = %xlate(lower: UPPER: %str(env));
   endif;

The getenv() routine returns a pointer to a C-style string, so I use the %STR built-in function to convert it to an RPG varchar variable. I’m also using the %XLATE built-in function to convert from lowercase to uppercase to ensure that the data will always be uppercase when I use it later in the code.

The next thing it will need is to get the customer number out of the URL (if any was supplied). The HTTP server provides the URL in an environment variable named REQUEST_URI that can be retrieved with the following code:

   dcl-s url varchar(1000);

   env = getenv(‘REQUEST_URI’);
   if env <> *null;
      url = %xlate(UPPER: lower: %str(env));
   endif;

The result is a variable named “url” that I’ve also converted to all uppercase. From that URL, I can retrieve the customer number by looking for it after the string “/rest/custdetail/”. Since I know “/rest/custdetail/” will always be in the URL (unless something went wrong), I can just scan for it and use that to get the position where the customer number is found. If there is no customer number, I’ll simply set my internal customer id to 0 and a code later in the program will use that to determine if a customer number was provided.

   dcl-c REQUIRED_PART const(‘/rest/custdetail/’);
   dcl-s pos int(10);
   dcl-s custpart varchar(50);
   dcl-s custid packed(5: 0);

   monitor;
      pos = %scan(REQUIRED_PART:url) + %len(REQUIRED_PART);
      custpart = %subst(url: pos);
      custid = %int(custpart);
   on-error;
      custid = 0;
   endmon;

Now that I know the method and customer number, I can choose what to do with it.  The code to handle GET is different than the code to handle PUT, for example, so I will select which section of code to run using a SELECT/WHEN group.

   select;
   when method = ‘GET’ and custid = 0;
      // code to list all customers
   when method = ‘GET’;
      // code to retrieve one customer
   when method = ‘PUT’;
      // code to update a customer (idempotent)
   when method = ‘POST’;
      // code to write a new customer (non-idempotent)
   when method = ‘DELETE’;
      // code to delete a customer
   endsl;

I will not provide all the code to do each of these options in the blog post, but instead at the bottom of this article because most of it is just standard RPG code that reads and writes data from the database. If you do want to see the program in its entirety, I will provide a link at the end of the article that you can use to download all the source code.

Working with JSON Using YAJL

The other part of the RPG code that I’d like to explain in this article is the code that works with JSON data. Although the DATA-INTO opcode may make this simpler in the future, as I write this, I haven’t had a chance yet to try it out. What I can show you is the “old fashioned” way of using YAJL to read and JSON data in an RPG program.

Although there are other ways to read JSON, I recommend using the “tree” style approach. I’ve found this to be the easiest, and in my RPG tools for YAJL, I provide a routine called yajl_stdin_load_tree that gets the data from the HTTP server and loads it into a YAJL tree in one subprocedure call.

   dcl-s docNode like(yajl_val);
   dcl-s errMsg varchar(500);

   docNode = yajl_stdin_load_tree(*on: errMsg);
   if errMsg <> ;
      // JSON was invalid, return an error message
   endif;

Now the JSON data has been loaded into a “tree-like” structure that is inside YAJL. The docNode variable is a pointer that points to the “document node”, which is a fancy way of saying that it points to the { and } characters (JSON object element) that surrounds the entire document. I can use the yajl_object_find procedure to locate a particular field inside that object and return a pointer to it. For example:

   dcl-s node like(yajl_val);

   node = yajl_object_find(docNode: ‘errorMsg’);
   if node = *null;
      cust.errorMsg =;
   else;
      cust.errorMsg = yajl_get_string(node);
   endif;

In this example, I retrieve a pointer to the value of the “errorMsg” field that is inside the JSON object. If “errorMsg” was not found, the pointer will be set to *NULL, in which case I know there wasn’t an error message sent. If it was sent, I can use the yajl_get_string subprocedure to get an RPG character string from the pointer.

In this case, the errorMsg field will be a character string, but other JSON fields might be numeric or boolean. I can retrieve these by calling other YAJL routines, such as yajl_get_number, yajl_is_true, and yajl_is_false in place of the yajl_get_string in the preceding example.

When a subfield has been set to an object or array, there are several different options to process it. I’ve already shown an example of yajl_object_find to process a single field inside of an object, but in addition to that, there are the following routines:

  • yajl_object_loop = loops through all of the fields in an object, one at a time
  • yajl_array_loop = loops through all of the elements in an array, one at a time
  • yajl_array_elem = get a particular array element by its index

Since this article is already very long, I will not provide examples of these in the post. Instead I will refer you to the link at the end of the article where you can download the complete example.

Once you’ve read your JSON data into your RPG program, you’ll want to ask YAJL to free up the memory that it was using. Remember, YAJL loaded the entire document into a tree structure, and that is taking up some storage on your system.  t’s very easy to free up the storage by calling yajl_tree_free and passing the original document node.

   yajl_tree_free(docNode);

YAJL will also be used to generate the JSON document that is sent back. For example, when you ask for the details of a customer, it uses YAJL to generate the response. The code in RPG looks like this:

   yajl_genOpen(*on);
   yajl_beginObj();

   yajl_addBool(‘success’: cust.success);
   yajl_addChar(‘errorMsg’: cust.errorMsg);

   if cust.success = *on;

      yajl_beginObj(‘data’);

      yajl_addNum(‘custno’: %char(cust.data.custno));
      yajl_addChar(‘name’: %trim(cust.data.name));

      yajl_beginObj(‘address’);
      yajl_addChar(‘street’: %trim(cust.data.address.street));
      yajl_addChar(‘city’:   %trim(cust.data.address.city));
      yajl_addChar(‘state’%trim(cust.data.address.state));
      yajl_addChar(‘postal’: %trim(cust.data.address.postal));
      yajl_endObj();

      yajl_endObj();

   endif;

   yajl_endObj();

   if cust.success;
      yajl_writeStdout(200: errMsg);
   else;
      yajl_writeStdout(500: errMsg);
   endif;

   yajl_genClose();

Prior to running this routine, the RPG program has loaded all the customer information into the “cust” data structure. The preceding routine uses that database data together with YAJL’s generator routines to create a JSON document.

The yajl_genOpen procedure starts the YAJL generator. It accepts one parameter, which is an indicator that determines whether the generated document is “pretty” or not. When the indicator is on, it will format the data with line feeds and tabs so that it’s easy for a human being to read. When the indicator is off, it will use the smallest number of bytes possible by removing line feeds tabs and extra spacing so that the entire JSON document is one big line of text. The “not pretty” version is a little more efficient for the computer to process, but since it is harder to read, I typically pass *ON while testing my program and change it to *OFF for production use.

When the JSON document calls for a new object (identified by the { and } characters), you can create it by calling yajl_beginObj. This will output the { character as well as make any subsequent fields be subfields of the new object. When you’re done creating the object, you can use yajl_endObj to output the } character and end the object.

There is a similar set of routines named yajl_beginArray and yajl_endArray that create a JSON array (vs. an object) that I did not use in this example but are useful when an array is called for.

Adding a field to an object or array is done by calling the yajl_addNum, yajl_addChar and yajl_addBool routines for numeric, character and boolean JSON fields, respectively. These routines not only add the fields, but they take care of escaping any special characters for you so that they conform to the JSON standard.

A really good way to understand the generator is to compare the expected JSON document (as I described in the section titled “The Customer Details Example” above) to the statements that generate it. You’ll see that each time a new object is needed, it calls yajl_beginObj, and each time a field is added to an object, it calls the appropriate “add” routine for the data type. These statements all correspond exactly to the JSON data that is output.

Once I’ve called the YAJL routines to add the data, I’ll have a JSON document, but it will be stored internally inside YAJL’s memory. To write it out to the HTTP server (which, in turn, will send it on to the program that is consuming our REST API), I use the yajl_writeStdout routine. This routine lets me provide an HTTP status code and an error message as a parameter. I recommend using a status of 200 to indicate that everything ran correctly or 500 to indicate that an error occurred.

Finally, once the JSON document has been sent, I call yajl_genClose to free up the memory that was used by the JSON document that YAJL generated.

Conclusion

YAJL is a powerful way to write JSON-based REST APIs. Although some of the examples may seem a little confusing at first, once you’ve successfully written your first one, you’ll get the hang of it quickly. Then it’ll be no problem to write additional ones.

You can learn more about YAJL and download the code samples that I mention in this post from my website at:

www.scottklement.com/yajl

If you’ll be at COMMON’s PowerUp18 conference, be sure to check out my session, Providing Web Services on IBM i, where I will demonstrate these techniques as well as other ways to provide your own web services in RPG.

Unlock Your RPG Applications to Enable Modern Online Experiences

By Dan Magid

What programming languages or frameworks do you think of when you hear web development? Probably JavaScript; maybe PHP or Python. But RPG? Not likely.

Nonetheless, for those who rely on longstanding IBM i systems to support their mission-critical operations, accessing current RPG applications is a necessity for powering fully functional, modern mobile applications. Of course, you might ask how a 60-year-old programming language could possibly handle the demands of modern online interactions.

RPG wasn’t originally built to power the web, but believe it or not, it may just be your fastest, most cost-effective option. With technological advances, it’s not always about what something was decades ago, but rather what it can do for you today.

In fact, RPG-based applications are being repurposed in ways their original developers could not have imagined, with amazing results. There is a significant amount of domain knowledge about the unique way your company does business built into your RPG applications. With the rise of the API economy, developers need that functionality for web, mobile and other applications. While RPG itself was not built with the web in mind, evolutionary technology has turned RPG into a web powerhouse, enabling the creation of web and mobile UIs—and even completely reengineered workflows—using the capabilities of existing RPG applications. And, RPG programs can call APIs from virtually any other application. Now, RPG code from 40 years ago underpins some of the most innovative applications you’ll see for call center and customer self-service applications, for example.

But why go this route when you could replace your old applications with pre-packaged or custom alternatives? The answer is that building upon what you already have—your venerable RPG applications— is faster, lower risk and significantly less expensive than replacing an entire system. Remember that those applications often have decades of business rules that must be replicated by any new system you buy or build from scratch, and that the TCO of your IBM i is on average about 1/3 of the most popular web and mobile platforms.* By keeping the back-end RPG application in place and using APIs or modern UIs, businesses can deliver renewed value from their current investments while still offering the kinds of features and experiences employees and customers need.

 

* Quark + Lepton, 2017 Study

Guest Blogger

Daniel Magid is Managing Director of Rocket Software’s Application Lifecycle Management & DevOps lab, and is a recognized authority on helping leading organizations achieve compliance through ALM solutions and DevOps best practices. He has written a variety of articles for leading IT publications and is a regular speaker at technology conferences.

Completely Free ILEditor and IBM Technology Refresh Recap

Today I’ll look at a powerful open source (and completely free!) IDE for ILE programs (CL, C/C++, Cobol or RPG) named ILEditor that is being actively developed by Liam Allan who is one of the brightest minds in the industry. In fact, last week Allan added a new GUI interface to the editor that makes it feel much more professional, while keeping it easy to use. I’ll also give you a quick overview of the announcement IBM made last week about updates to IBM i 7.2 and 7.3.

The IBM Announcement

On February 13th, just in time for Valentine’s Day (because IBM wants to be my valentine!), IBM announced new Technology Refreshes. These include support for POWER9 processors, which look incredible – but, alas, I’m not a hardware guy. They also include updates to Integrated Web Services (IWS), Access Client Solutions (ACS), RPG and more.
Here are links to the official announcements:

IBM i 7.2 Technology Refresh 8 (TR8)

IBM i 7.3 Technology Refresh 4 (TR4)

You should also check out Steve Will’s blog post.

My Thoughts

The most exciting part of this announcement for me is the introduction of the new DATA-INTO opcode in RPG. Here’s the sample code that IBM provided in the announcement:

DATA-INTO myDs %DATA(‘myfile.json’ : ‘doc=file’) %PARSER(‘MYLIB/MYJSONPARS’);

It appears that this will work similarly to Open Access, where the RPG compiler will examine your data structure and other variables that it has all the details for and work together with a back-end handler that will map it into a structured format. Open Access refers to the back-end program as a “handler”, whereas DATA-INTO seems to call it a “parser”, but the general idea is the same.

As someone who has written multiple open source tools to help RPG developers work with XML and JSON documents, this looks great! One of the biggest challenges I face with these open source projects is that they don’t know the details of the calling program’s variables, so they can’t ever be as easy to use as a tool like XML-INTO. For example, the YAJL tools that I provide to help people read JSON documents require much more code than the XML-INTO opcode, because XML-INTO can read the layout of a data structure and map data into it, whereas with YAJL you must map this data yourself. However, DATA-INTO looks like it will solve this problem, so that once I’ve had time to write a DATA-INTO parser, you’ll be able to use YAJL the same way as XML-INTO.

Unfortunately, as I write this, the PTFs are not yet available, so I haven’t been able to try it. I’m very excited, however, and plan to blog about it as soon as I’ve had a chance to try it out!

What is ILEditor?

ILEditor (pronounced “I-L-Editor”) came from the mind of Liam Allan, who is one of the best and the brightest of the 2018 IBM Champions. I have the privilege of working with Liam at Profound Logic Software, and I can tell you that his enthusiasm for computer technology and IBM i programming know no bounds. In fact, one day last week after work, Liam sent me a text message about his new changes to ILEditor, sounding very excited. When I factored in the time zone difference, I realized it was 1:00 a.m. where he lives!

For many years, one of the most common laments in the IBM i programming community has been about the cost and performance of RDi. Please don’t misunderstand me. I love RDi, and I use it every day. I believe RDi is the best IDE for IBM i development that’s available today. That said, sometimes we need something else for various reasons. Some shops can’t get approval for the cost of RDi. Others might want something that uses fewer resources or something they can install anywhere without needing additional RDi licenses. Whatever the reason, ILEditor is very promising alternative! I wouldn’t be surprised if it eventually is able to compete with RDi.

Why Not Orion? Or SEU?

The concept of Orion is a great. It’s web-based, meaning that you don’t have to install it and it’s available wherever you go. Unfortunately, it’s not really a full IDE – at least not yet! I hope IBM is working to improve it. It does not know how to compile native ILE programs or show compile errors. Its interface is designed around the Git version control software, which makes it tricky to use unless you happen to store your code in Git. And quite frankly, it’s also a little bit buggy. I hope to see improvements in these areas, but right now it’s not a real option.

The most popular alternative to RDi today is SEU. In fact, historically this was the primary way that code was written for IBM i. So, you may think it’s still a good choice. However, I don’t think it’s viable today for two reasons:

  1. The green-screen nature makes it cumbersome to use. This is no problem for a veteran programmer, because they’re used to it. But for IT departments to survive, they need to bring in younger talent. Younger talent is almost always put off by SEU. I even know students who gave up the platform entirely because they thought SEU seemed so antiquated, and they wanted no part of it.
  2. SEU hasn’t received any updates since January 2008. That means all features added to RPG in the past 10 years – which includes three major releases of the operating system –will show as syntax errors in SEU.

About ILEditor

ILEditor is open source, runs on Windows and was released as open source under the GNU GPL 3.0 license. That means it is free and can be used for both private and commercial use. If you like, you can even download the source code and make your own changes. It can read source from source members or IFS files. In addition to editing the source, it can compile programs, show you the errors in your programs, work with system objects and display spooled files. It even has an Outline View (like RDi does) that will show you the variables and routines in your program.

The main web site for ILEditor is: worksofbarry.com/ileditor/.

If you want to see the source code, you’ll find the Github project here.

You do not need to install any software on your IBM i to use ILEditor. Instead, the Windows program uses the standard FTP server that is provided with the IBM i operating system to get object and source information and to run compile commands. An FTPES (FTP over SSL) option is provided if a more secure connection is desired.

Connecting for the First Time

When you start ILEditor, it will present you with a box where you can select the host to connect to. Naturally, the first time you run it there will be no hosts defined, so the box will be empty. You can click “New Host” to define one.

Once you have a host defined, it will be visible as an icon, and double-clicking the icon will begin the connection.

When you set up a new system, there are five fields you must supply, as shown in the screenshot below:

Alias name = You can set this to whatever you wish. ILEditor will display this name when asking you the host to connect to, so pick something that is easy to remember.

Host name / IP address = the DNS name or IP address of the IBM i to connect to.

Username = Your IBM i user profile name.

Password = Your IBM i password – you can leave this blank if you want it to ask you every time you connect.

Use FTPES = This stands for FTP over Explicit SSL. Check this box if your IBM i FTP server has been configured to allow SSL and you’d like the additional security of using an encrypted connection.

The Main IDE Display

Once you’ve connected, you’ll be presented with a screen that shows the “Toolbox” on the left and a welcome screen containing getting started information and developer news, as shown in the screenshot below.

Any of the panels in ILEditor, including these two, can dragged to different places on the display or closed by clicking the “X” button in the corner of the panel. There is also an icon of a pin that you can click to toggle whether a panel is always open or whether it is hidden when you’re not using it. If you look carefully on the right edge of the window, you’ll see a bar titled “Outline View”. This is an example of a hidden panel. If you click on the panel title, the panel will open. If you click the pin, it will stay open. You can adjust the size of any panel by dragging its border.

When you open source code, it will be placed in tabs in the center of the display (just as the welcome screen is initially.) These can also be resized or moved with the mouse. This makes the UI very flexible and simple to rearrange to best fit your needs.

The Toolbox

Perhaps the best place to start is with the toolbox.  Here’s what that panel looks like:

Most of the options in this panel are self-explanatory. I will not explain them all but will point out a few interesting things that I discovered when using ILEditor:

  • The “Library List” is primarily used when compiling a program. This is the library list to find file definitions and other dependencies that your program will need.
  • The “Compile Settings” lets you customize your compile commands. Perhaps you have a custom command you use when compiling. Or perhaps you use the regular IBM commands but want to change some of the options used. In either case, you’ll want to look at the Compile Settings.
  • As you might expect, “Connection Settings” has the host name, whether to use FTPES and other settings that are needed to connect to the host. In addition to that, there are some other useful options hidden away in the connection settings:
    • On the IFS tab, you’ll find a place to configure where your IFS source code is stored and which library it should be compiled into.
    • On the Editor tab, there is a setting to enable the “Outline View”. You’ll want to make sure this is checked, otherwise you’ll be missing out on this feature.
    • On the ILEditor tab, there’s a setting called “Use Dark Mode”. This will change the colors when it displays your source code to use a black background (as opposed to the default white background), which many people, myself included, find easier on the eyes.
  • When you change something in the “Connection Settings” (including the options described above), you will need to disconnect from the server and reconnect so that the new settings take effect.

Opening Source Code from a Member List

ILEditor allows you to open source code from either an IFS file or a traditional source member. You can use the Member Browser or IFS Browser options in the toolbox to browse your IBM i to find the source you wish to open and open it.

The Member Browser opens as a blank panel with two text fields at the top. At first, I wasn’t sure exactly what these were for as there wasn’t any explanation. I guessed that this was where you specified the library (on the left) and the source physical file (on the right) that you wanted to browse. Iit turned out that I was correct. If you type the library and filename and click the magnifying glass, it will show you all the members in that file.

I have a lot of source members that I keep in my personal library, and I often get impatient waiting for the member list to load in RDi. I was pleasantly surprised to see that the member browser in ILEditor loads considerably faster.

There is also a “hidden” feature where you can press Ctrl-P to search the list of recent members that you listed in the member browser. Just press Ctrl-P and start typing, and it’ll show the members that match the search string. This was a very convenient way to find members.

Once you’ve found the member (in either the regular member browser or the “search recent” dialog), you can double-click on the member name to open it.

Create or Open a Member Without Browsing

In the upper-left of the ILEditor window, there is a File menu that works like the file menus found in most other Windows programs. You can click File/New to create a new member or IFS file or File/Open to open an existing member or IFS file when you know the name and therefore don’t need to browse for it.

The File Menu also offers keyboard shortcuts to save time. You can press Ctrl-O for Open, or Ctrl-N for New to bypass the menu.

One thing that I found a little unusual is that you must specify the source type when you open an existing member. I expected this when creating a new member, since the system doesn’t know what it is. But when opening an existing member, I expected it to default to the source type of the member so that you don’t have to specify it every time. I discovered that if you do not specify the type, it will default to plain text. I spoke to Liam about this, and he assured me that this is something he plans to improve in the future. Thankfully, this is not the case when using the member browser. It only happens when opening the member directly.

Working with IFS Files

The IFS Browser can be used to browse the IFS on your IBM i and find the source code that you’d like to open. It will begin browsing the IFS in the directory that you’ve specified in the IFS tab in your connection settings. Any subdirectories found beneath that starting directory can be expanded as well to see the files inside of it.

Like the member browser, double-clicking on an IFS file will open it in the editor.

The File menu also has options for creating a new IFS file or opening an existing IFS file when you know the exact path name. In that case, you do have to type the entire IFS path. There is no option to browse folders as you’d find in the open dialogs of other Windows software. That didn’t seem like a problem to me. If I wanted to see the folders, I’d use the IFS browser instead.

The Source Editor

I found the editor to be very intuitive, since it works the same as you’d expect from a PC file editor. It provides syntax highlighting and an outline view that make the source code very easy to read. In the screenshot below, I’m using “dark mode”, so you’ll see that my source code has a black background.

 

Syntax highlighting worked very nicely in free format RPG, CL and C/C++ code, including code that used the embedded SQL preprocessor.

Unfortunately, it did not work in fixed format RPG code. Liam tells me that fixed format RPG is especially difficult to implement because he codes ILEditor’s syntax highlighting using regular expressions, and regular expressions are difficult to make work for position-dependent source. However, he assured me that he does plan to support fixed format RPG code and is working on solving this problem.

I noticed that I could still type fixed format code and make changes to it, and aside from the source not being colored correctly, it worked fine.

The Outline View was a pleasant surprise, because I wasn’t really expecting an editor other than RDi to have one. It does not have as many features as the RDi outline view, but it worked very nicely for what I needed it for. I was also pleasantly surprised that the Outline View worked with CL code.

Compiling Programs

The compile option can be run by using the Compile menu at the top of the screen, the compile icon (shown in the picture below) or by pressing Ctrl-Shift-C.

I discovered that the compile option does not ask for any parameters. Instead, it uses the options that you specified in your connection and compile settings options in the toolbar. So if you want to change one of the default compiler options, you need to change them in the compile settings each time.

There are advantages and disadvantages to this approach. The advantage is that it’s very quick and easy to compile a program. When you’re developing software, you often have to compile it many times, and it’s very nice to be able to skip the dialog and just have it compile. The disadvantage is when you want to do something different in a one-off situation. You have to go into the compile settings to change it, so that’s a little bit of extra work. However, I find that I don’t need to do that very often, so this wasn’t a big deal to me.

When an error occurs during the compile, an error listing will open showing you what went wrong, very similar to what you’d find in RDi. Like RDi, you can click on the error and it will position the editor to the exact line of code where the error was found.

One thing that surprised me about the compile and the error message dialog was that it is considerably faster than RDi. That seems strange to me, since both tools are connecting to the IBM i and running the same IBM compiler for RPG. However, I found that depending on the size of the member, the ILEditor compile was 10-20 seconds faster than the RDi one.

RPG Fixed Format to Free Format Converter

One feature of ILEditor that simply did not work well was the RPG converter. Some of the fixed format code in my program would convert, but other things (including things that should’ve converted easily) did not. Code that spanned multiple lines did not convert at all.

In my opinion, the converter needs a lot of work before it will be useful. I pointed this out to Liam, and he told me that he agrees and has a complete rewrite of the converter on his to-do list.

Other Features

I’d like to mention some of the other features of ILEditor that I did not have time to try out before writing this article. Since I didn’t have time, I can’t review them and give my opinion – but, I wanted to mention them. That way, if you’re looking for these features, you can give them a try yourself and see what you think.

  • Source Diff = compares two sources (members or IFS files) and highlights what is different about them.
  • Spooled File Viewer = Lets you view spooled files that are in an output queue
  • SQL Generator = Generates SQL DDL code from an existing database object
  • Offline mode = lets you download source from the IBM i to store on your PC and work on it while you are not connected (for example, when traveling on a plane or train without good internet access), uploading the results later.

My Conclusion

I was extremely impressed by ILEditor. RDi has more features, such as debugging, refactoring and screen/report design, but I was surprised at just how many features ILEditor has, considering it was written by one man in his free time and costing nothing. I was pleasantly surprised by the performance of ILEditor, which was consistently faster than RDi while using far less memory.

Unfortunately, the lack of syntax highlighting for fixed format RPG will be a problem for many RPG developers, and I sincerely hope that does not discourage them from at least trying ILEditor.

If a lot of people try it, and some of them donate money or give their time to help with development, this tool could easily become a serious competitor to RDi.

A Powerful Way to Run Unix and Open Source Tools from a Program

Perhaps the biggest area of growth in IBM i programming over the past several years has been the Open Source languages. There are thousands of utilities, mostly designed for Unix, that you can run in the QShell and PASE environments, and these have become very popular on IBM i! However, running these tools from your RPG and CL programs can be tricky. This post will introduce you to a free utility called UNIXCMD that makes it much easier.

Why is it Tricky?

It’s tricky because there’s a difference in the way IBM i and Unix systems run programs. When a program is called on Unix, a new “process” is created, this is very much like a new job on IBM i, except that it is created each time a program is called. This gives the calling program a choice, it can either stop and wait for the called program to finish, or it can continue and run simultaneously with the program it called. In a way, this is similar to submitting a batch job on i, but it’s different in that both programs can interact with the user’s display.

There are two common ways of running Unix programs, PASE via the QP2SHELL API and QShell via the STRQSH (or it’s alias QSH) CL command.

The QP2SHELL API runs a PASE program directly in the current job, which will cause problems if the job expects to be able to run simultaneously with the caller. Programs that use multiple threads, in particular, can have strange problems that are very hard to troubleshoot. The fix is to spawn a child job to run QP2SHELL so that it’s not in the same job as the caller. To enable input and output, you need to connect pipes to that child job. This is a lot of work, and often more than a programmer bargained for. (In fact, my description is somewhat oversimplified, to keep this brief!)

The STRQSH CL command solves this by always spawning its own child job and connecting pipes to it. It does all of this for you. The problem with this is that you are limited in how you can interact with the input and output streams. For a program to work with them, the only real option is to read and write from temporary files. This works, but it cumbersome, you have to create the file, clear it, redirect the I/O, and can only read the output once the whole process has finished.

Scott’s Solution

My solution is to create tools that do the work of submitting the child job for you, and connecting the pipes to a simple interface that is easy to use from your program. For a CL program, I’ve provided simple commands that open, read, write and close the connection, allowing you to read and write in a natural way. In RPG, I’ve used the Open Access interface so that you can open the connection with RPG’s native file interface, and read and write using the standard open, read, write and close opcodes.

The RPG Interface

Let’s take a look at RPG first. My initial examples use the newer “all-free” approach to writing RPG, but if you’re unable to use newer RPG, don’t fret – see the section titled “Older RPG Code”, below.
This example switches to the /QIBM directory in the IFS, lists the files in that directory, and prints them to the spool:

Notice the HANDLER keyword on the DCL-F statement. This tells RPG to access this file through the Open Access interface. When you run any of RPG’s file opcodes against this file, it will (under the covers) call routines in the UNIXCMDOA program, that allows my tool to take control and handle all of the work for you.

The command to run is provided in the second parameter to the HANDLER keyword. Since I’m setting that command in calculations in my program, I do not open the file until the command variable is set. For that reason, the file is declared with the USROPN keyword, and I open it explicitly with the OPEN opcode.

Unix utilities allow you to run multiple commands on a single line if you separate them with a semicolon. In this example, the cd command is used to switch to the /QIBM directory, and then the ls command is run afterwards to get the output.

To get the output from the Unix ls (list directory) command, I simply use the READ opcode. Since this is a program-described file (there are never any defined fields in a Unix input or output stream) I am using RPG’s feature that lets me read program-described data into a data structure.

When I’m done, I use the CLOSE opcode to shut down the Unix process. One thing that surprises people who are new to UNIXCMD is that any errors that occur in the background Unix program will be reported on the CLOSE opcode rather than the OPEN, READ or WRITE. This is because the Unix program has the opportunity to write error messages, and will not report that it has failed until the program has ended. Catching errors can be done easily with RPG’s MONITOR and ON-ERROR opcodes.

By default, the UNIXCMD utility runs your program using the QShell interface. If you prefer to run the PASE interface and avoid the QShell environment, you can do that by prefixing the command string with “pase:”, as shown in the next example.

The examples so far have only read output from a Unix command. In the next example, I’d like to demonstrate sending data both ways. In this case, I’m calling a PHP script that calls a web service to Geocode and address. In other words, I pass an address as input, and the script returns the latitude and longitude coordinates where that address can be found. To do that, I’ve written a PHP script that receives the address from it’s “standard input” (that is, the pipe that is connected to its input stream) and write the coordinates to standard output. (If you’re interested in the PHP code, it is included in the downloadable examples on my web site.) To call it from RPG, I can simply write my data to it, and then use the READ opcode to get the results.

The first thing you’ll notice in this statement is that it sets the Unix PATH variable. This is done because many people don’t have the PHP command in their PATH. PATH is very much like a library list, except that it is a list of IFS directories that the Unix environment uses to find a command. The PATH statement adds the directory where we’ve placed the php-cli command (CLI stands for “command line interface”) so that QShell can find it.

Another important thing to note is that data sent as input through the pipe is not automatically converted from EBCDIC to ASCII or Unicode. To solve that problem, I added a call to the QShell iconv utility, which can translate between different CCSIDs. In this case, will convert between 0 (a special value that means “this job’s CCSID”) and iso-8859-1 which is CCSID 819 and is a flavor of ASCII.

A Note About Input and Output

UNIXCMD assumes that you will write all the data that is sent as input to the Unix command first, before the first time you read its output. When you read the Unix output, it will shut down the Unix input stream to signal the Unix program that no more data is coming. This works well in most applications.

However, if you would prefer that it not close the stream, this can be made with a very simple code change to the UNIXCMD utility. If this would be useful to you, please e-mail me at commonblog@scottklement.com and I’ll be glad to show you how to change it.

Older RPG Code

Sadly, not everyone has a current version of RPG. To help those people, I’ve provided a way of using UNIXCMD that’s compatible with even the oldest versions of RPG IV using the SPECIAL file interface. Here is the last example rewritten to use that approach.

Notice that the command is passed to the SPECIAL file through the PLIST. The second parameter in the PLIST is not shown in this example, but you can add a 1 character second parameter and set it to P if you want to run in the PASE environment. If you do not pass this parameter (or set it to a Q) it will run QShell instead. Here is an excerpt of code that does that:

Since the SPECIAL file approach does not require Open Access, it will work all the way back to V5R3.

Using the UNIXCMD Tool from CL

Since the CL programming language does not support Open Access, there is no way to use the standard IBM supplied SNDF or RCVF opcodes that you are used to using for a normal file. Instead, I have created my own CL commands named OPNPIPE, SNDPIPE, RCVPIPE and CLOPIPE that handle the open, send, receive and close functions, respectively. The OPNPIPE command accepts the command to run, and lets you designate whether it is PASE or QShell. Aside from these differences, the UNIXCMD utility works the same from CL as it does from RPG.

Here’s an example of listing files (like the first RPG example) using the CL interface:

Get the UNIXCMD Utility

UNIXCMD is an open source tool that is available at no charge. You can download it, the examples given in this article, and a few more examples from my web site at the following link:

www.scottklement.com/unixcmd

IBM i and the API Economy

By Dan Magid

The API economy is a new frontier for companies seeking to get the most out of their data. With the right API, you can transform rigid workflows into the intuitive web-based and mobile experiences that today’s users demand, using modern programming languages to access real-time transactional data. This way, users can harness the value of information from the transactional systems that fuel their businesses without unnecessary time, cost, and risk.

According to Coleman Parkes Research, 88% of all businesses employ APIs. But our internal research shows that only 38% of IBM i users are enjoying the benefits of the API economy. Perhaps this disparity is due to the lack of API expertise among IBM i shops, where most developers use RPG and COBOL vs. modern tools like Node.js or GO, or the time and resource constraints that are common among IBM i shops.

However, the fact is that APIs are the most effective way for IBM i shops to expose real-time transactional data for access by modern web, mobile and IoT applications, so bridging the knowledge gap between modern application development and the languages that power your critical IBM i applications should be a top priority. But it must be done without hurting productivity, blowing the IT budget or risking catastrophic application errors.

These goals—building and deploying APIs without negatively affecting organizational performance—may appear to be contradictory, but they don’t have to be. How do you do it? I recommend a two-pronged approach:

  • Look for tools to build APIs directly from the host-based application that run your business—but without needing to modify the codebase that runs your business
  • Ensure that your API functionality is easily tested against any other host-based application changes that occur, vastly reducing the resources required to ensure that APIs continue to work in a world of ever-changing user and IT requirements

Following this approach will enable you to not only extend the value of your IBM i platform, but also unlock the value of your applications to a world of new uses. And without the need to modify source code, it will also help you bridge the knowledge gap between your IBM i applications the web and mobile applications that need their functionality.

I’ll finish with the results of a recent study by Quark and Lepton, which showed that the IBM i platform typically costs about one-third as much budget to operate over a three-year period compared to Oracle/Linux or SQL Server/Windows platforms. IBM i is by far the most cost-effective, and with the right API approach, it can serve you effectively for years to come.

Guest Blogger

Daniel Magid is Managing Director of Rocket Software’s Application Lifecycle Management & DevOps lab, and is a recognized authority on helping leading organizations achieve compliance through ALM solutions and DevOps best practices. He has written a variety of articles for leading IT publications and is a regular speaker at technology conferences.

The Python Programming Language

Python is a powerful, interpreted, open-source language available for most operating systems, including IBM i. It has many uses and has a very easy to learn syntax. Many packages are available to add capabilities not included in the Python core. Python is a great language to use for general purpose programming of all sorts.

Open Source - Python

 

 

 

 

Python Language Details

Python structure uses white space instead of brackets. This makes the code more readable than in similar languages such as Perl. The syntax is logical and easy to follow, even for beginners, and typing is dynamic. Even with this simple syntax, the language also has great flexibility and powerful capabilities. Object-oriented programming is possible but not required, as it is in other languages like Java or Ruby.

Web Development

One use of Python is for back-end web development. Frameworks such as Flask and Django simplify back-end development using built-in components that are popular. Rapid development and prototyping are possible using these frameworks. Python is also good for general scripting, including projects involving crawling the web or scraping information from websites.

Data Science

Python is a great language to use for data science. Useful libraries include NumPy, which adds speedy mathematical capabilities, and SciPy, which adds the ability to manipulate complex data in arrays for linear algebra and other purposes. Several other libraries are useful for doing data science in Python.

Philosophy

Python has a philosophy often referred to as “The Zen of Python“. This philosophy encourages simple, clean code that is easy to read and explain to others. It has several other suggestions for optimizing the use of Python to make it uniform and ideal using best practices.

Summary

A simple yet powerful open-source language, Python is a great choice for many projects. Using Python gives you many capabilities out of the box and additional packages to add more. The language is quite readable, and its uses include many technical domains.

Learn Python with COMMON’s Open Source Video Series.

 

Python

Wide Open

From the Vault – The Computing of Business, October 2014

EarthquakeDid you know that in the wide world of open source software there is an application for analyzing seismic data? If I had only known that a few weeks ago I could have thrown a portable seismograph into my carry-on for a recent trip to California.

Since I was rudely awakened during my stay by what FEMA now calls a major earthquake, I could easily have done a quick data reduction and submitted the results to the open source Global Quake Model. But alas, during the event I was thinking more about what might happen to the roof than contributing to science. At 40 miles from the epicenter it was certainly a unique experience and the building held together nicely—check that one off the bucket list.

So open source is everywhere. Good developers are developing good, creative software and distributing for all to use within the confines of a number of different license platforms.

In a recent IT-related university classroom experience Phil McCullough, a fellow COMMON Director, noted that the entire curriculum centered on open source. Open source operating systems, open source databases, open source development tools, open source applications; the entire gamut. The message was loud and clear: open source will be a big part of our computing future.

But software, by its very nature, contains defects. In many cases those defects do not consist of what we traditionally consider a problem. For example, as the programmable point of sale system industry was developing a couple of decades ago, I cannot imagine that leaving a credit card number unencrypted in ram memory would ever be considered a problem. Turns out it was.

Recently my employer sold some software we had developed to a very large company. Parts of that software contain open source components like frameworks and other mechanisms that helped develop a very complex set of code. Before the deal could close the buyer required substantial documentation of every open source component, and the version of that component. Additionally, the entire base of object code was scanned for any known vulnerabilities. All went well and the deal was completed.

Open source operating systems like Linux, and large open source applications like Sugar CRM, have the benefit of having substantial support organizations. Dedicated people watching for problems are an advantage many small applications do not, and cannot, possibly have. The likelihood is very real that the bad guys—and there are an awful lot of them—will stumble on to something they can exploit in almost any open source project. If you run some or all of a smaller targeted code base, it then becomes simply a matter of a bad guy (they are voracious sharers of exploit information) finding your system and deploying one of the myriad ways of injecting malware on to it.

Of course there needs to be something worthwhile to steal. Credit card numbers are the currency de jour though certain kinds of pictures bring a bigger bang. Regardless, any information about your company or your customers has value to someone.

A number of scanning solutions like OpenLogic and OpenVAS  can attempt to locate problematic software. But using such tools seems to me to be a bit reactive. The veritable cat may have already left its bag.

A more proactive approach is to know where in your software, and on your systems, the open Ground Crackssource stuff (and anything else with known vulnerabilities) is. Trust me, the stuff is there. Complete, accurate application inventories are more important than ever. An inventory, coupled with appropriate monitoring of the threat landscape, will keep you ahead of the bad guys. Yes, this is something more to do with your constantly shrinking resource base. But staying ahead of the bad guy sure beats the alternative.

For many the 2014 South Napa earthquake was a disaster. For me it was an interesting experience.

For some the open source landscape will be a disaster. Reading about those problems should be the only experience you want.

About the Author: Randy Dufault, CCBCP

Randy DufaultRandy is the Director of Solution Development for Genus Technologies, a Midwestern consultancy dealing primarily with enterprise content management systems. His experience with content management dates back 25 years, where he helped develop what ultimately became IBM‘s Content Manager for iSeries. He has also developed and integrated a number of advanced technologies including document creation, character recognition, records management, and work flow management. Randy is a member of the COMMON North America Board of Directors and was active in the development of COMMON‘s Certification program.

Read Randy’s Computing of Business column in COMMON.CONNECT.

The Ruby Programming Language

RubyRuby is an open source language that is available for the IBM i operating system. While its syntax is very similar to other open source languages commonly used for the web, such as Python and Perl, Ruby has many unique characteristics that make it a great choice for your project.

 Ruby’s Purpose

Like many modern languages, Ruby is dynamically typed and supports both object-oriented programming and other programming paradigms. While most commonly used in web development, Ruby is a general-purpose language used for different types of applications.

Object Based

According to this article on pocoo.org, Ruby differs from Python in that it is a pure object-oriented language where everything is an object. Tricks in using the objects allow the use of other programming paradigms. Python, on the other hand, has both functions and objects and is thus a hybrid language.

Design Philosophy

Ruby is a language that has a special philosophy behind it about design and usage. While designed as a fun and simple language for programmers to write in, Ruby also allows great complexity in the programs that it can create. Furthermore, Ruby is a very flexible language that allows developers a great amount of creativity in the implementation process.

Popularity

Ruby is quite a popular language and is very much in demand in the industry. Ruby on Rails is a framework that is commonly used in web development and greatly extends Ruby’s capabilities. Distributed under the MIT License, Ruby on Rails is a Model-View-Controller framework that integrates well with common web technologies such as JavaScript. This framework allows for the fast development of secure and scalable web applications.

Learn Ruby for IBM i – along with Node.js and Python – with COMMON’s Open Source Video Tutorials. Learn more.

Want more Open Source for IBM i? Check out these 2017 Fall Conference & Expo sessions.