Monitor and Respond to Conditions of Interest using Agent Templates

Instructions to have PacketWise automatically monitor any of a large set of potential problems and then optionally respond with notification or corrective actions

PacketWise's adaptive response feature automatically monitors for conditions of interest, detects potential problems, notifies somebody if a problem is detected, and takes corrective actions. You get to decide:

• what constitutes a problem
• if you want notification and, if so, by what method
• if a corrective action is needed and, if so, which actions are appropriate

For background on the adaptive response feature, including the definitions for agents, action files, red/yellow/green status categories, and more, see Adaptive Response Overview.

Phase One: Plan Your Adaptive Response Agent

1. Peruse the list of adaptive response agent templates to find the one that interests you. Start a page of notes for your agent plan. Record the agent name as the first item on your plan.
   
   There are several groups of Packeteer-provided adaptive response agents that you can customize, including categories for the health of the network, the unit itself, and applications.
   
   If there is not a Packeteer-provided agent that monitors for the condition you have in mind, you can create your own agent using any of the over 100 metrics PacketWise measures and stores. If you want to create a metric-based agent, consult the Monitor and Respond to My Own Custom Condition recommendation instead of this one.
   
   We'll use the following scenario as an example throughout the rest of the steps in this recommendation:

Suppose you want to detect any user who uses a really excessive amount of bandwidth. You'd like to drastically limit their access to network resources, at least for the rest of the day. You decide that although you'd like all users to stay below 5 percent of capacity, you won't take action unless someone exceeds 12 percent. (If you are really interested in implementing this scenario, see Quarantine Bandwidth Abusers for the full solution.)

For our example, you notice that the agent template called High Bandwidth Host looks useful, and you jot its name on your plan.

2. Check to see if you can have multiple instances of the agent that interests you. This information is in the same agent template list you used in the previous step.
   
   You can have only one instance of some types of agents, and you can have multiple instances of others. For example, the agent that checks for bandwidth-greedy hosts can have only one instance. That makes sense because you certainly don't need to check for greedy users twice at the same time. But the agent that checks on retransmissions can have multiple instances. This also makes sense because you can check one, some, or many different traffic classes for individual levels of retransmissions.
   
   If there can be multiple instances, and there is not already an agent monitoring the target you have in mind for your condition, then you will add an agent. Otherwise, you will edit an agent. Make a note on your plan as to whether you will add or edit your agent.
   
   
3. Explore the information about your chosen agent from the same agent template list you used in previous steps. Then determine the agent thresholds and parameters you'd like to use.
   
   What condition indicates a problem or a need to respond (the red threshold)? What should indicate that things are okay (the green threshold)? Are the agent's default values for the red and green thresholds reasonable for your needs?
   
   For our example, you want to catch users whose usage exceeds 12 percent of capacity, and the default red value is 10. So, you note that you'll need to change that. You decide that you'd like users to stay below 5 percent of capacity, and the default green value is indeed 5. No change will be needed there.
   
   
4. What actions do you want to take when your problem gets bad (crosses the red threshold)?
   
   Do you want to notify somebody? Do you want to execute some other commands to document or even correct the problem?
   
   An action file is essentially a script that contains the actions you decide are appropriate. An action file can be any sequence of PacketWise CLI commands. PacketWise automatically executes the commands in your action file when your threshold is crossed. You don't necessarily need to take any action. Many times, just being able to check if your condition transpired is enough.
   
   Each agent has a set of variables whose values are passed along at run time to any triggered action file. The action file can use any of those variables' values, or not (your choice). You might think of the variables as parameters to the action file to avoid confusion with PacketWise measurement variables. Consult the Action Files Overview to see which variables (or parameters) your agent makes available.
   
   For our example, if a host exceeds 12 percent of capacity, you want to cap their access in the future. That will be done by moving their traffic into a traffic class with per-user limits. Again, the Quarantine Bandwidth Abusers recommendation has all the details. But for our purposes, the task you need to enforce when a user gets greedy is to add their IP address to an existing host list called Abusers.
   
   Continuing our example, you note that $host-ip is one of High Bandwidth Host's variables. So your red action will indeed have access to the greedy host's IP address in order to add it to the host list.
   
   
5. Decide on an appropriate action or notification (if any) for when your problem returns to normal levels and stops being an issue. Again, the action file can be any sequence of PacketWise CLI commands and can use any of the variable values made available by the agent.
   
   For our example, it would seem that when a formerly greedy bandwidth abuser's usage declines to 5 percent of capacity, you would want to stop restricting their access. But, that is probably not a great idea.
   
   You are limiting your greedy user's access, so of course usage would drop dramatically. If you restore full access as soon as usage drops, your abuser would be liberated almost immediately, and that wouldn't make for a very effective deterrent. So you will have to attend to the task of freeing greedy users somewhere else (other than in a green action).
   
   Make sure you think through your own situation's planned actions with the same amount of scrutiny we did with the sample scenario. Automatic responses to detected conditions are powerful, but use their power wisely to avoid any potential snafus.
   
   
6. How often would you like PacketWise to automatically check for your condition, called an evaluation interval? Check the agent's default value in the same agent template list you used in previous steps. Make a note of the interval you want.
   
   For our example, the default interval is just a minute, and that's fine. But if, for example, it were one week instead, then that might be a problem. A greedy user could wreak a lot of havoc and impact many others in a week's time before being caught.
   
   
7. Make a note of any other details or tasks to which you must attend, if any, to complete your plan.
   
   For our example, you need to figure out how you will restore users to the network's good graces. Because you decided not to do so in the green action, where should it happen? You have a couple choices.
   
   You could add another command to the red action file. It could schedule (using the CLI schedule command) the future removal of the greedy host from the Abusers host list. Alternatively, you could automatically clear the Abusers host list at the end of each day. We'll choose the first alternative here because it uses an adaptive response feature, the focus of this recommendation.
   
   
8. With your notes intact, proceed to the next phase.
   

Phase Two: Create your Action Files

You can create your agent and action files in either order. You'll create your action files next here because it's convenient to have your action file names when you define your agent.

1. You can use three styles of notification (email, SNMP trap, syslog) in your action files. Each requires some setup before the corresponding CLI command will succeed. If you chose any notification to be included in either action file, attend to the required setup, as follows:
   
   
   • If you want to be notified via SNMP traps, configure PacketWise for SNMP support and define the SNMP trap destination(s).
   • If you want to be notified via email, set up email notification by configuring an SMTP server and defining email recipients.
   • If you want a message to be sent to a syslog server, set up the syslog feature.
     
     
2. If your plan calls for a red action, you must create the action file that will be executed automatically when your red threshold is crossed.
   
   Consult Create a Command File, Action Files Overview, and PacketWise CLI commands for extra information you might need to create your action file.
   
   For our example, you would do the following:
   • Using the Info tab's File Browser, navigate to 9.258/Agent/Cmd.
   • Click new cmd file and enter a name in the File Name field: BADHOST.
   • Enter your script in the Contents area of the Command File Editor. It would look like this:
     #Title: Restrict a greedy host by moving it to the Abusers host list (and remove it later)
     hl add abusers $host-IP
     schedule new today 23:55 "hl delete abusers $host-IP"
     
     
3. If your plan calls for a green action, you must create the action file that will be executed automatically when your green threshold is crossed (from a yellow or red status).
   
   Consult Create a Command File, Action Files Overview, and PacketWise CLI commands for extra information you might need to create your action file.
   
   For our example, you would not create a green action file.
   
   
4. Test your action files with the run button, as described in Create a Command File, supplying sample values for any $variables you used in your script of CLI commands. PacketWise automatically prompts you for sample values and, once entered, displays the output in the results area of the page.
   
   
5. Note the names and spellings of your two action files on your plan notes. You will need them later when modifying your agent.
   
   

6. If you want to use the same action file in multiple agents, but are stumped because you would need different variables/parameters in the script's CLI commands on different occasions, you can still pursue your plan. You will need to define your own parameters in your action file. Basically, you define a parameter by using its name preceded with a $ in your action script. Then, you supply a value when you define your agent.
   
   For example, suppose you want to create one action file that sends an email. You'd like to use the same action file for at least five different agents. But you want to send the email to different recipients, depending on the agent. The application manager needs the email from agents monitoring applications, and the network manager needs the email from agents monitoring network functions. You'd enter the CLI command to send email:
   send email $emailaddress $subject_heading $message_to_admin
   and, in doing so, define three new user-defined variables. Later, when you edit your agents that reference your action file, you'll supply the parameter values for any user-defined variables.
   
   The Action Files Overview has explanations and examples of user-defined variables/parameters.

Phase Three: Add or Edit your Agent

1. Consult your plan to see if you need to add or edit your agent. Accordingly, add an agent or edit an agent.
   
   Supply a name for your new agent, if you are creating a new one.
   
   
2. Fill in the agent's parameters with values from your plan notes.
   
   Revise the red or green thresholds, if the values in your plan are different than the values already there. If you defined red and green action files, enter the filenames in the appropriate fields.
   
   Note: It can be prudent and less risky to first define and test your agent without the names of your action files (so that actions won't be automatically triggered before you are confident of your agent). Test your agent and your action files separately, then you can add the action file names to the agent later.
   
   Revise the evaluation interval field if the value in your plan is different than the value already in the agent.
   
   
3. If there are any parameters listed that need values, supply them.
   
   For example, if you are editing an agent that monitors the efficiency of a particular traffic class, then you need to enter the traffic class name. If you defined any user-defined variables in your action file (as described in the last step of the previous phase, above), you need to supply the values here.
   
   
4. Save your new agent's definition.
   
   
5. Check your agent's status occasionally and check its reports.

See also:

Adaptive Response Agent Examples