Project Title: Automated STROBE batch measurement reports and alerts

Subject: Instead of ad-hoc or permanent monitoring, an individual solution based on Compuware’s STROBE will periodically monitor distinct address spaces at distinct time and duration. Measurement reports will be created automatically from STROBE sample datasets. A REXX procedure will then analyse measurement reports. In case of breaking threshold values, a recipient list will receive e-mails showing indication to specific performance indicators.

Solution: Compuware STROBE, JCL, REXX procedure, job scheduler (e.g. TWS).

Part I – initialization, creation of STROBE measurement tasks

//*********************************************************************
//* INITALIZE STROBE BATCH FOR PARTICULAR CICS ADDRESS SPACES,
//* 30 MINUTES, 10.0000 SAMPLES, EXPECT 1 DATASET FOR EVERY ADDRESS 
//* SPACE AND MEASUREMENT.
//*********************************************************************
//STROBINI EXEC PGM=STRBCSR
//SYSPRINT DD SYSOUT=*
//SYSIN   DD *
DELETE MYCICS1
DELETE MYCICS2

DELGRP MYGRP01

ADDGRP MYCICS1,STEP=*,GOMIN=30,SAMPLES=10000,GROUP=MYGRP01,
DSNAME=MY.STROBE,UNIT=SYSDA,NONOTIFY,
DISPOSITION=CATLG,LIMIT=(1,QUIT),
DB2=(SQLATTR,CAPTURE),CICS=(RLI),COBOL,PLI,MQSERIES,
ISPFFLAG=4002

ADDGRP MYCICS2,STEP=*,GOMIN=30,SAMPLES=10000,GROUP=MYGRP01,
DSNAME=MY.STROBE,UNIT=SYSDA,NONOTIFY,
DISPOSITION=CATLG,LIMIT=(1,QUIT),
DB2=(SQLATTR,CAPTURE),CICS=(RLI),COBOL,PLI,MQSERIES,
ISPFFLAG=4002

Part II – plan/initiate STROBE daily measurement, executed 2 times a day

//*********************************************************************
//* 1 X DAILY 2 MEASUREMENTS PER DAY,
//* SET/CHANGE CURRENT DATE YYMMDD IN STROBE SAMPLE OUTPUT DATASETS,
//* THEN PUT MEASUREMENT INTO THE QUEUE.
//*********************************************************************
//STROBESM EXEC PGM=STRBCSR
//SYSPRINT DD SYSOUT=*
//SYSIN   DD *
CHGGRP MYCICS1,GROUP=MYGRP01,SAMDSN=&TWSDSLVL..STROBE.D&OYMD1
CHGGRP MYCICS2,GROUP=MYGRP01,SAMDSN=&TWSDSLVL..STROBE.D&OYMD1

SUBMIT MYGRP01,
STARTDATE=('&CDATE','&CDATE'),STARTTIME=('09:30','15:00')
/*

Replace variable variable &TWSDSLVL. by an adequate high index level. Replace &OYMD1 by current (julian or ISO) date. Replace &CDATE by current date. Expect a sample dataset for every addresse space and for every measurement (e.g. at 9:30 am and 3:00 pm).

Once the online component of STROBE has executed a measurement request, it creates a STROBE sample dataset with a unique dataset name. STROBE option SAMDSN=&TWSDSLVL..STROBE.D&OYMD1 results in &TWSDSLVL..STROBE.D&OYMD1..MYCICS1.S001D001 for the first request for „MYCICS1“. Users will need to specify this output dataset name within the next step.

Part III – creation of a STROBE measurement from sample datasets

//*
//*********************************************************************
//* CREATE A REPORT USING 1ST STROBE SAMPLE DATASET,
//* REPLACE &OYMD1 IN DS-NAME UND SYSIN CARD BY DATE (JULIAN FORMAT).
//*********************************************************************
//*
//STROSAMP SET STROSAMP=&TWSDSLVL..STROBE.D&OYMD1..MYCICS1.S001D001
//STRPRF SET STROPROF=&TWSDSLVL..STROBE.D&OYMD1..MYCICS1.REPORTAM
//*
//*%OPC BEGIN ACTION=NOSCAN
//*
//*********************************************************************
//DELPROF  EXEC PGM=IEFBR14
//SYSUT1   DD DSN=&STROPROF,
//            DISP=(MOD,DELETE),
//            SPACE=(TRK,(1,1),RLSE)
//ALLOPROF EXEC PGM=IEFBR14
//SYSPRINT DD DSN=&STROPROF,
//            DISP=(NEW,CATLG,DELETE),
//            UNIT=SYSDA,
//            SPACE=(CYL,(3,10),RLSE)
//*********************************************************************
//PROFILE  EXEC PGM=STROBEDT
//SYSPRINT DD DSN=&&PROF121,DISP=(NEW,PASS,DELETE),
//            SPACE=(CYL,(3,10)),
//            UNIT=SYSDA
//SYSUT1   DD UNIT=SYSDA,SPACE=(CYL,(3,10))
//STRSAMPL DD DISP=SHR,
//            DSN=&STROSAMP
//STRMAP   DD DUMMY,DCB=BLKSIZE=1692
//SYSUDUMP DD DUMMY
//SYSTERM  DD SYSOUT=*
//SYSOUT   DD SYSOUT=*
//SORTWK01 DD UNIT=SYSDA,SPACE=(1700,(3000),,CONTIG)
//SORTWK02 DD UNIT=SYSDA,SPACE=(1700,(3000),,CONTIG)
//SORTWK03 DD UNIT=SYSDA,SPACE=(1700,(3000),,CONTIG)
//SORTWK04 DD UNIT=SYSDA,SPACE=(1700,(3000),,CONTIG)
//SORTWK05 DD UNIT=SYSDA,SPACE=(1700,(3000),,CONTIG)
//SORTWK06 DD UNIT=SYSDA,SPACE=(1700,(3000),,CONTIG)
//SYSTMP01 DD UNIT=SYSDA,SPACE=(3400,(6000),,CONTIG)
//SYSTMP02 DD UNIT=SYSDA,SPACE=(3400,(6000),,CONTIG)
//SYSTMP03 DD UNIT=SYSDA,SPACE=(3400,(6000),,CONTIG)
//USERMAP  DD DUMMY
//*%OPC END   ACTION=NOSCAN
//SYSIN    DD *
REPNAME='PLX1 MYCICS1 &CYYYY-&CMM-&CDD',LINEMAX=60,
NODSCS,WAITLOC,NOATTR=VSAM
/*
//*%OPC BEGIN ACTION=NOSCAN
//*********************************************************************
//REFORMAT EXEC PGM=STROBEDS
//SYSPRINT DD DSN=&&PROF121,DISP=(MOD,DELETE,DELETE)
//PROFILE  DD DISP=MOD,
//            DSN=&STROPROF
//SYSOUT   DD SYSOUT=*
//SYSUDUMP DD DUMMY
//*%OPC END   ACTION=NOSCAN
//*
//XMPSTRCK EXEC PGM=IKJEFT1A,PARM='XMPSTRCK'
//STEPLIB  DD DSN=&TWSDSLVL..LOADLIB,DISP=SHR
//SYSEXEC  DD DSN=&TWSDSLVL..SYSEXEC,DISP=SHR
//SYSTSPRT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//IN       DD DSN=*.REFORMAT.PROFILE,
//            DISP=SHR
//OUT      DD DISP=(,PASS),DCB=(LRECL=80),
//            UNIT=SYSDA,SPACE=(TRK,(15,15),RLSE)
//SYSTSIN  DD DSN=NULLFILE
//THRESHLD DD *
#SUS;10.00;3.00;;TOTAL
#WTM;10.00;0;;TOTAL;SVC 001
#PSU;35.00;0;;TOTAL
#PUP;20.00;0;;TOTAL
#CSS;10.00;3.00;;TOTAL
//*
//CHECK    IF XMPSTRCK.RC = 4 THEN
//*------------------------------------------------------------------*
//* SEND AN EMAIL TO RECIPIENT LIST, E.G. DBA'S
//*------------------------------------------------------------------*
//MAIL     EXEC PGM=MAIL,
//         PARM=('XMPSTRCK MYCICS1 STROBE REPORT ANALYSE')
//SYSPRINT DD SYSOUT=*
//SMTPMAIL DD SYSOUT=(B,IZTIP3SM)
//NOTIFY   DD DSN=&TWSDSLVL..MAIL.LISTS(DBALIST),
//            DISP=SHR
//MAIL     DD DSN=*.XMPSTRCK.OUT,
//            DISP=(OLD,DELETE)
//         DD *
THIS MAIL HAS BEEN GENERATED AUTOMATICALLY BY JOB MYJOB1.
PLEASE DO NOT RESPOND.
//CHECK    ENDIF

This job consists of mainly 3 steps: Create and format an STROBE measurement report using an STROBE sample dataset. It uses the STROBE batch utilities STROBEDT and STROBEDS. Then next step then executes the REXX procedure XMPSTRCK to check specific areas of the report for threshold violations. In case of a finding it ends with RC=4. The next step will then send the REXX report as an email to a recipient list.

To individualize the reports a bit, users may change the report name using the option REPNAME=’PLX1 MYCICS1 &CYYYY-&CMM-&CDD‘. (PLX1=LPAR in this example, MYCICS1=address space invastigated, other variables for example result in 2013-12-05).

The first sample started at 09:30 will be reported into dataset &TWSDSLVL..STROBE.D&OYMD1..MYCICS1.REPORTAM. The second sample started at 15:00 (3 pm) creates a sample dataset ending with S002D001. Users need to run a second job to create a report which dataset name ends REPORTPM.

Part IV – houskeeping – delete obsolete datasets

//*
//*-------------------------------------------------------------------*
//* JI: LISTC+SORT+DEL = DELETE OBSOLETE STROBE REPORTS OLDER 45 DAYS
//*-------------------------------------------------------------------*
//LISTC   EXEC PGM=IDCAMS
//SYSPRINT DD DISP=(,PASS,DELETE),DCB=(RECFM=VBA,LRECL=137),
//            UNIT=SYSDA,SPACE=(TRK,(3,10))
//SYSOUT   DD SYSOUT=*
//SYSIN    DD *
 LISTC LEVEL(&TWSDSLVL..STROBE) CREATION(45)
/*
//*  //*
//*  //PRINTLC   EXEC PGM=IEBGENER     * FOR DEBUG ONLY
//*  //SYSUT1   DD  DSN=*.LISTC.SYSPRINT,DISP=(OLD,PASS)
//*  //SYSUT2   DD  SYSOUT=*
//*  //SYSIN    DD  DUMMY
//*  //SYSPRINT DD  SYSOUT=*
//*
//SORTDS   EXEC PGM=SORT
//SYSPRINT DD  SYSOUT=*
//SYSOUT   DD  SYSOUT=*
//SORTIN   DD  DSN=*.LISTC.SYSPRINT,DISP=(OLD,PASS)
//SORTOUT  DD  DISP=(,PASS,DELETE),DCB=(RECFM=FB,LRECL=67),
//             UNIT=SYSDA,SPACE=(TRK,(3,10))
//SYSIN    DD  *
 OPTION COPY
 INCLUDE COND=(6,7,CH,EQ,C'NONVSAM')
 OUTFIL  FNAMES=SORTOUT,VTOF,
         BUILD=(C' DELETE ',22,45,C' PURGE SCRATCH')
 END
/*
//*  //*
//*  //PRINTSR   EXEC PGM=IEBGENER     * FOR DEBUG ONLY
//*  //SYSUT1   DD  DSN=*.SORTDS.SORTOUT,DISP=(OLD,PASS)
//*  //SYSUT2   DD  SYSOUT=*
//*  //SYSIN    DD  DUMMY
//*  //SYSPRINT DD  SYSOUT=*
//*
//DELETE  EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSOUT   DD SYSOUT=*
//SYSIN    DD DSN=*.SORTDS.SORTOUT,DISP=(OLD,PASS)

The job determines and deletes STROBE sample and report datasets older than 45 days.

Report examples

###############################################################################
# XMPSTRCK - STROBE report analytics - V1.3              13/07/31 10:30:37 #
###############################################################################

STROBE Report IN-DSN ..: MYUSER.STROBE.D130731.MYCICS2.REPORTAM

-------------------------------------------------------------------------------
##### Finding on line 50142:
#CSS                     ** CPU USAGE BY SQL STATEMENT **
STMT  STATEMENT           STMT  EXECUTION    % CPU TIME  MARGIN OF ERROR   .00%
NUMBER   TEXT               CNT   AVG-TIME          TOTAL  .00     2.00     4.00
5465 OPEN   408DECLAR         1   4.466389           .06  .
-------------------------------------------------------------------------------

section  ! total > ! exec avg > ! alt. symbol =  ! ignore str =   ! findings
---------!---------!------------!----------------+----------------+---------
#SUS     !   10.00 !   3.000000 !                ! TOTAL          ! 0
#WTM     !   10.00 !   0.000000 !                ! TOTAL SVC 001  ! 0
#PSU     !   35.00 !   0.000000 !                ! TOTAL          ! 0
#PUP     !   20.00 !   0.000000 !                ! TOTAL          ! 0
#CSS     !   10.00 !   3.000000 !                ! TOTAL          ! 1

***end***

###############################################################################
# XMPSTRCK - STROBE report analytics - V1.3              13/07/22 14:30:08 #
###############################################################################

STROBE Report IN-DSN ..: MYUSER.STROBE.D130722.MYCICS1.REPORTPM

-------------------------------------------------------------------------------
##### Finding on line 995:
#PUP                     ** PROGRAM USAGE BY PROCEDURE **
SYSTEM      SYSTEM SERVICES              .COBLIB      COBOL LIBRARY SUBROUTINE
MODULE  SECTION   FUNCTION                   % CPU TIME  MARGIN OF ERROR 23.10%
IGZCPAC  IGZCIN1   (V3) INSPECT             22.22  22.22  .*******************
-------------------------------------------------------------------------------

section  ! total > ! exec avg > ! alt. symbol =  ! ignore str =   ! findings
---------!---------!------------!----------------+----------------+---------
#SUS     !   10.00 !   3.000000 !                ! TOTAL          ! 0
#WTM     !   10.00 !   0.000000 !                ! TOTAL SVC 001  ! 0
#PSU     !   45.00 !   0.000000 !                ! TOTAL          ! 0
#PUP     !   20.00 !   0.000000 !                ! TOTAL          ! 1
#CSS     !   10.00 !   3.000000 !                ! TOTAL          ! 0

***end***

###############################################################################
# XMPSTRCK - STROBE report analytics - V1.3              13/08/02 10:30:33 #
###############################################################################

STROBE Report IN-DSN ..: MYUSER.STROBE.D130802.MYCICS1.REPORTAM

-------------------------------------------------------------------------------
##### Finding on line 17320:
#PUP                     ** PROGRAM USAGE BY PROCEDURE **
SYSTEM      SYSTEM SERVICES              .NUCLEUS     MVS NUCLEUS
MODULE  SECTION   FUNCTION                   % CPU TIME  MARGIN OF ERROR  5.14%
IEAVEWAT           WAIT SERVICE             13.46  21.70  .************++++++++
-------------------------------------------------------------------------------
##### Finding on line 47876:
#CSS                     ** CPU USAGE BY SQL STATEMENT **
STMT  STATEMENT           STMT  EXECUTION    % CPU TIME  MARGIN OF ERROR   .00%
NUMBER   TEXT               CNT   AVG-TIME          TOTAL  .00     1.00     2.00
5465 OPEN   299DECLAR         2   4.126343           .09  .*
-------------------------------------------------------------------------------

section  ! total > ! exec avg > ! alt. symbol =  ! ignore str =   ! findings
---------!---------!------------!----------------+----------------+---------
#SUS     !   10.00 !   3.000000 !                ! TOTAL          ! 0
#WTM     !   10.00 !   0.000000 !                ! TOTAL SVC 001  ! 0
#PSU     !   35.00 !   0.000000 !                ! TOTAL          ! 0
#PUP     !   20.00 !   0.000000 !                ! TOTAL          ! 1
#CSS     !   10.00 !   3.000000 !                ! TOTAL          ! 1

***end***