1. Highlights of Cheryl Watson’s TUNING Letter 2006 No. 5
2. Red Alert for HSM Users on z/OS 1.8
3. Google Source Code Search
4. UIC Update Performance Problem
5. Free SMF Reference Summary
1. Highlights of Cheryl Watson’s TUNING Letter 2006 No. 5
The forty-seven page 2006 No. 5 TUNING Letter was emailed to subscribers on October 24, 2006. Single issues may be obtained for $135 each from our Web site at http://www.watsonwalker.com. The following is a summary of just some of the contents of this latest TUNING Letter:
Workload Manager Update
A good portion of this issue consists of our SHARE Trip Report, which starts on page 26. While attending SHARE, we always pay close attention to the sessions related to the Workload Manager. You can use this z/OS component to define application priorities, and then the Workload Manager attempts to schedule your work based on those instructions. Because the happiness of your users and the satisfaction of your customers depend on the Workload Manager operating correctly, it is important that you understand it and be aware of any potential problems or announced enhancements. Please make sure to see our Focus article on the Workload Manager, which starts on page 16. It consolidates much of the good Workload Manager information we obtained from SHARE and our own research. If you’re planning to install a zAAP or a zIIP, there is a useful table that helps you keep track of the latest Workload Manager enhancements related to those specialty processors. There is also a handy table that helps you correlate recent enhancements with the various z/OS releases. Finally, we describe Resource Group enhancements that are available to those running z/OS 1.8.
Avoiding Cybercrime
The FBI defines a Cybercrime as any situation where a computer is used as a tool to commit a crime, or as the target of a crime. Anyone who has used a computer has probably received one or more phony emails, trying to get you to divulge personal information or buy questionable products. But does anybody really fall for these scams, and aren’t they just pulled off by kids with too much free time? If you think that Cybercrime is not a big deal, then please enlighten yourself by reading our Fighting Cybercrime article on page 32. You’ll be surprised at the scope of this problem, and the sophistication of cyber criminals. Failure to understand these techniques could put you and your organizations in jeopardy. We provide some useful online resources to help educate you, and to help you fight back.
Important Maintenance
We try to fill every TUNING Letter issue with news about important software maintenance that you should investigate. These are recommendations that are passed along by IBM and other readers who have experienced problems. Avoiding just one unscheduled service outage per year will save you much more than the cost of your subscription, so we hope you pay attention to these. Starting on page 35, we describe two new Red Alerts from IBM. On page 36 you can read about an SRM overhead problem that has been discovered on the new System z9 machines. Finally, we pass along our usual recommendations in the Interesting APARs section on page 4.
Elsewhere in this Issue
Users of the CA-NeuMICS product should read page 7 to learn about a useful tool that will soon be available for no extra charge. • Even if you don’t have a membership in SHARE, you can still browse the newly-available Bit Bucket Archive (see page 27). • Do you often use Adobe Reader or Adobe Acrobat to browse IBM manuals? If so, then please see page 37 to learn about a free plug-in that will make your searches much more effective.
2. Red Alert for HSM Users on z/OS 1.8
IBM will issue a Red Alert when it finds a software problem that can be particularly disruptive or destructive. The previous issue of Cheryl’s List documented four Red Alerts that were opened in July and August of this year. Since that issue, another Red Alert was opened on October 13, 2006. It describes a potential data loss problem that can occur with the HSM RECYCLE function when running under z/OS 1.8.
Here is a summary of the problem, as taken from the Red Alert:
A problem was detected in the V1R8 DFSMShsm RECYCLE function which under certain circumstances, can potentially result in data loss. The problem occurs only when the RECYCLE function encounters certain errors with its input tape. The most common type of error would be the failure to mount the input tape, resulting in the failure of the RECYCLE function with message ARC0833I return code 31. The problem detected causes DFSMShsm to lose knowledge of the logical EOF (End of File) on the output tape. As a result, the next time that DFSMShsm attempts to write to the recycle output tape, it will write from the load point, overwriting data previously written to the tape. Attempts to recall or recover the overwritten data sets will fail with message ARC1001I return code 68 reason code 16.
THE PTF for APAR OA18465/UA29904 will cause RECYCLE processing to maintain the logical EOF information for its output tape. Error scenarios that would have caused DFSMShsm to lose knowledge of the logical EOF on the recycle output tape will now result in an ABEND of the RECYCLE task. Although the attempt to recycle the input tape will fail, the potential for the output tape being subsequently overwritten is eliminated.
V1R8 customers are strongly advised to avoid running the RECYCLE function on V1R8 DFSMShsm until the PTF for APAR OA18465/UA29904 has been installed. Please see APAR OA18465 for more information.
The DFSMShsm development team has created a tape assessment tool for those customers that have run RECYCLE already. Information on the tool can be found in APAR OA18465.
It is recommended that customers DO NOT run RECYCLE on a V1R8 DFSMShsm prior to installing PTF UA29904 for OA18465. All installations that have already done this are requested to mark any partial ML2 and backup tapes full using the ‘DELVOL volser MIGRATION(MARKFULL)’ or ‘DELVOL volser BACKUP(MARKFULL)’ commands, as appropriate.
DFSMShsm RECYCLE can be run safely on systems with z/OS V1R7 and below.
Here is more information about the APAR that is described in the Red Alert, as well as information about a second APAR that is referenced by the first one. Both of these fixes are marked as HIPER (High Impact, Pervasive):
- OA18372 (DFSMS for z/OS 1.8, 20Oct2006) – Data Loss after RECYCLE Error RECALL and RECOVER Fail with IOS000I EOD and ARC1168I RC16. HSM RECYCLE processing can cause the overwriting of a migration or backup tape if the recycle that most recently used the tape for output failed with message ARC0833I and return code 31. Attempts to RECALL or RECOVER data sets that were overwritten will fail with message ARC1001I or message ARC1168I. Depending on the type of tape device, it is possible that some data sets that were not overwritten can be successfully retrieved. HSM has been modified to correct the problem.
- OA18465 (DFSMS for z/OS 1.8, 12Oct2006) – Relief for OA18372. HSM RECYCLE processing will be changed to call error-processing module ARCERP to abnormally terminate under the conditions that would otherwise result in the loss of data described in APAR OA18372. Although the RECYCLE function may then terminate abnormally under certain error conditions, customers can resume normal recycle processing without fear of data loss. OA18372 will provide a complete resolution to the problem that will eliminate the deliberate ABEND conditions added by this fix. The APAR text also gives instructions for downloading and using a tool that can be used to detect potential damage caused by the original problem.
Use the Web link http://www.ibm.com/servers/eserver/support/zseries/ to view the current Red Alerts. Just select ‘Red Alerts for System z’ from under the ‘Popular links’ heading to display the current entries. Once you are on this page, you can also register so that you will be sent an email when a new Red Alert is issued. We strongly believe that at least one person from every organization should be subscribed to this service.
3. Google Source Code Search
Another search option recently added by Google is the ability to search for source code found on publicly-accessible Internet sites. This is just like the regular Google search, except that it has options that restrict the search to sites that contain source code. You can access the feature using the linkhttp://www.google.com/codesearch. You can then just type in your search argument, press the “Search Code” button, and let Google do the rest.
At a test of the feature, we searched for the term “GETMAIN” to find samples of assembler language code that used this common macro. This search resulted in approximately 20,600 hits. Most of them had nothing to do with assembler language, but were C/C++ programs that had user-defined methods or variables with names such as “GetMainXml.”
In an attempt to refine our search, we used the “Advanced Code Search” function that can be selected from the starting search page. This allowed us to enter our search argument (GETMAIN), and to also use a pull-down list to select the language we wanted to find (Assembly). More than 30 languages are supported, from Ada to Yaac. Another advanced option allows you to look for code that is supported under a specific software license (such as GNU General Public License or NASA Open Source Agreement). You can also restrict your search to package names or file names that match patterns you specify. You can also specify whether the search should consider case sensitivity or not.
Our second attempt (using filtering by language) was more successful, returning approximately 50 hits. Most of these did appear to be assembly language programs, although sometimes the word GETMAIN appeared as a comment in the source code, rather than as a code statement.
From the Advanced Code Search screen, you may also want to select the link entitled “About Google Code Search.” This provides more information about this new service, and contains the answers to some frequently asked questions. One of the things we learned here is that you can do filtering from the regular code search page by using operators as part of the search argument. For example, specifying “lang:java” will restrict the search to programs written in the Java language. By placing a minus sign in front of the operator, you request that no matches be returned that are found in that language. For example, “-lang:java” would not return any search hits found within Java source modules.
We suspect that this resource will be most useful to those who program in higher-level languages such as C++ and Java (the option to filter by COBOL is not even supported). But based on our limited testing, there does appear to be some code available in the public archives that can run on the z/OS platform. If the amount of this code increases over time, this new code search feature will be even more valuable. There is an “Add Code” option that allows you to specify the URL of a site containing source code that Google will then index and add to its source code archive. The linkhttp://www.google.com/codesearch/addcode can be used to request this function. Perhaps those who maintain Web sites containing public z/OS source code will consider doing this so that all of us will be able to find the code sample that we need.
4. UIC Update Performance Problem
In our previous TUNING Letter, we documented a potential performance problem relating to the way UIC update is done for large address spaces (see TUNING Letter 2006 No. 4, page 26). Here is the APAR we included that documented this problem:
- OA16097 (z/OS 1.5+, OPEN 17Aug2006) – UIC Update during Swap in of a Very Large Logically Swapped Address Space Can Take Seconds. During the swap in of a logically swapped address space, module IRARMSWP calls the UIC update routine to process the frame queues for the address space. The parameter list passed to UIC update contains the maximum values for the time to stay disabled and the number of frames to process per queue. Module IRARMSWP uses a hard-coded value of x’7FFFFFFF’ for the maximum time to stay disabled. In one reported case, the address space being swapped in owned many data spaces and was using six GB of central storage. The large maximum time value caused the system to remain disabled until the entire address space was processed by UIC update, which took over nine seconds. The SRM lock was held during this time period and the other processors on the system were spinning during this time and waiting for the SRM lock to be released. The local fix is to make large address spaces non-swappable, so they will not be subject to swap in or swap out processing.
We suspect that many of our readers noted this problem, but didn’t do much about it because a fix was not available. The APAR was finally closed on September 28, 2006, and the PTFs were made available on October 10, 2006.
Also, IBM has recently updated the APAR to add the HIPER and PERFORMANCE flags. This indicates a potential performance problem that could have very negative side effects. Users who have experienced the problem have reported that their systems seem to freeze up for several seconds at a time. This can cause performance problems and potential application failures.
Now that fixes are available, we suggest you research this problem and determine whether you could be exposed to it.
5. Free SMF Reference Summary
For more than a decade, we have provided a free SMF Reference Summary that can be downloaded from our Web site (http://www.watsonwalker.com). We updated that summary in January of this year. The new version is eight pages in length, and includes material up through z/OS 1.7. For each SMF record type and subtype, we provide a short description, and the name of the SMF exit associated with the record (when documented). There is also a summary of SMFPRMxx parameters, a summary of supported SMF exits, a bibliography of IBM SMF reference manuals, and a sample job for dumping SMF records.
You can download the free reference by using the Web address shown above, selecting “Articles,” from the menu bar on the left, and scrolling to the bottom of the page. Or you can use the linkhttp://www.watsonwalker.com/SMFreference.pdf to access it directly.
We recently received an email from a user who was still using an old version of the reference (1995), and was wondering if a newer version was available. This made us realize that we probably could have done more to publicize this new version. Although we did notify our newsletter readers of the update (see TUNING Letter 2006 No. 2, page 35), this is the first time we have mentioned this useful resource in Cheryl’s List.
Although our Web site provides much information about all of the Cheryl Watson products (TUNING Letter, CPU Chart, BoxScore, GoalTender), we also provide some great resources that can be accessed for no charge by customers and non-customers alike. These include sample TUNING Letter issues, past presentations from SHARE and other conferences, and the popular Workload Manager Quickstart policy. Make sure to visit our Web site soon, and see if any of these resources can help you be more effective. If you are not currently a TUNING Letter subscriber, please read the sample issues. Because each one is 40+ pages in length, they provide much more useful technical information than can be provided via this short email.
Stay tuned!