Essay: Log analysis

CHAPTER 1
INTRODUCTION
1.1 Brief Introduction
Log analysis is an art and science seeking to make sense out of computer-generated records. Log Analysis is also referred to as system or network log analysis. The process of creating such records is called data logging. Logs are emitted by operating systems, network devices, applications and all manner of intelligent or programmable device .A stream of messages in time-sequence often comprise a log. Logs may be directed to files and stored on disk, or directed as a network stream to a log collector. Log messages must usually be interpreted with respect to the internal state of its source (e.g., application) and announce security-relevant or operations-relevant events (e.g., a user login, or a systems error).
Logs are often created by software developers. It is created to aid in the debugging of the operation of application .The syntax and semantics of data within log messages are usually application or vendor-specific. Terminology may also vary; for example, the authentication of a user to an application may be described as a login, a logon, an user connection or authentication event. Hence, log analysis must interpret messages within the context of an application, vendor, system or configuration in order to make useful comparisons to messages from different log sources. Log message format or content may not always be fully documented.
1.2 Motivation
Task of the log analyst is to induce the system to emit the full range of messages in order to understand the complete domain from which the messages must be interpreted. Through experience it makes the task of finding out the bugs within the log files much easier.
Similarly the products like W4N, ViPR, NCM, ECS, SRM, SMARTS etc. Also generates Log files which are to be analyzed by the log analyst to fix them at the e the bugs in the products in order to keep the product up and running without any interruption. If any bugs found it has to be fixed as soon as possible to avoid other malfunctions. To find out the bugs easily in the log files and to fix them at the earliest LOG ANALYZER was developed. This tool helps the analyst to find out the bugs within the log files in many ways, it also provides 3 main functionalities to locate a bug in the Log Files.
1.3 Scope
This project ensures finding bugs within the log files providing the below features:-
o Standalone Desktop Based Tool.
o Easy, Friendly User interface.
o Ability to perform search on single and multiple files.
o Support for Multiple Products: Logs from multiple products like W4N, ViPR, and NCM etc’can be searched within the framework.
o Supports Multiple File Type
o Search Entire Folder.
o Multiple Search Option.
o Multiple Options to Display Errors.
o Provide Highlighters for the Errors found.
o Panels in the UI should be Adjustable.
CHAPTER 2
LITERATURE SURVEY
Literature survey is carried out in order to analyze the background of the current project which helps to find out flaws in the existing system & guides on which unsolved problems can be workout. Log Analyzer tools are available in the market most of them are web based application with limited features. If Literature Survey is carried out correctly then it helps the developer in development process.
2.1 What is a Search Log?
A search log is a log file of the communications (i.e., transactions) between a system and the users of that system. Rice and Borgman (1983) present transaction logs as a data collection method that automatically captures the type, content, or time of transactions made by a person from a terminal with that system. Peters (1993) views transaction logs as electronically recorded interactions between on-line information retrieval systems and the persons who search for the information found in those systems. For Web searching, a search log is an electronic record of interactions that have occurred during a searching episode between a Web search engine and users searching for information on that Web search engine. A Web search engine may be a general-purpose search engine, a nice search engine, a searching application on a single Website, or variations on these broad classifications. The users may be humans or computer programs acting on behalf of humans. Interactions are the communication exchanges that occur between users and the system. Either the user or the system may initiate elements of these exchanges.
2.2 How are These Interactions Collected?
The process of recording the data in the search log is relatively straightforward. Servers record and store the interactions between searchers (i.e. ,actually Web browsers on a particular computer)and search engines in a log file (i.e., the transaction log) on the server using a software application. Thus, most search logs are server-side recordings of interactions. Major Web search engines execute millions of these interactions per day.
The server software application can record various types of data and interactions depending on the file format that the server software supports.
2.3 Why Collect This Data?
Once the server collects and records the data in a file, one must analyze this data in order to obtain beneficial information.
Few Tools are given below:-
o PowerGREP:- PowerGREP is a powerful Windows grep tool. Quickly search through large numbers of files on your PC or network, including text and binary files, compressed archives, MS Word documents, Excel spreadsheets, PDF files, OpenOffice files, etc. Find the information you want with powerful text patterns (regular expressions) specifying the form of what you want, instead of literal text. Search and replace with one or many regular expressions to comprehensively maintain web sites, source code, reports, etc. Extract statistics and knowledge from logs files and large data sets.
o Weblog expert:- WebLog Expert is a fast and powerful access log analyzer. It will give you information about your site’s visitors: activity statistics, accessed files, paths through the site, information about referring pages, search engines, browsers, operating systems, and more. The program produces easy-to-read reports that include both text information (tables) and charts. View the WebLog Expert sample report to get the general idea of the variety of information about your site’s usage it can provide.
o Log Parser Lizard:- Log Parser Lizard is a GUI for Microsoft Logparser, definitely the best one available on the market today. Log Parser is a very powerful and versatile query software tool that provides universal query access (using SQL) to text-based data, such as log files, XML files, and TSV/CSV text files, as well as key data sources on the Microsoft Windows operating system, such as the Windows Event Log, IIS log, the registry, the File System, the Active Directory services and much more.
o Piwik, Oracle log analyser, Wget are few other log analysis tool .
2.2 Existing System
Presently there is no specific tool available for to find exact bugs in the log file. Different Log Analysts use different method to find the bugs within the log file .One such Tool which they currently use is Note Pad++.Note Pad++ searches a file for multiple keywords. You can specify a list of keywords to search for in the current file, and filter out lines that match any of the keywords in this list. It was developed mainly for analyzing log files where you are interested in more than one keyword and the order in which they appear.
o Matching lines are listed with their line numbers in a separate panel in the plug-in window
o Double clicking a matched line in this panel will take you to the corresponding line in the original document
o Options to copy the filtered lines to clipboard and highlight matches in the original file
o Supports case sensitive search, whole word matching and regular expressions. Regexp is enabled by default
2.3 Problem Statement
Note Pad++ is not efficient in all ways.
o It only performs keyword search.
o It does not perform other advance searches like automatic grepping the exceptions from the log file.
o Fails to a load a log file more than few MB’s length.
o Using this tool is not that effective. Log Analyst must go through the log files line by line in order to fix a bug
o Time consuming
o If a larger log file has to be searched it has to be first split into many chunks, open each chunk and find the errors in it manually.
o To split the files some other tool should be used first and then open each split in Note Pad++ every time.
o If there are many larger files as such the time complexity increases
o If the issue is critical it has to be escalated within short amount of time ,if this is the case to split it and them find errors in each split it will affect the client’s environment who is waiting for the issue to get fixed.
2.4 Proposed System
The proposed system has a lot of new features which would help the log analysts perform the log analysis quickly and accurately. Following are the features of the Log Analyzer.
o Standalone Desktop Based Tool: The Tool will be a standalone desktop based application which helps the users to install them on Laptop or Personal computer. It is a light weight tool(uses minimum system resources)
o Easy, User Friendly User interface: The Tool has a very understandable GUI where Tool Tips for all the components will be provided to guide the user in the correct path in order to use the tool effectively and more efficiently.
o Ability to perform search on single and multiple files: When other tools have problem opening a single file to find errors this tool has a ability to perform search on multiple files on a single selection.
o Provides folder Search option: where a search error in a single large file is a challenge here an entire folder could be searched.
o Support for Multiple Products: Logs from multiple products like W4N, ViPR, and NCM etc’can be searched within the framework. This tool is not specific to a product.
o Supports Multiple File Type: The tool supports different files types. Following files are accepted as input file .txt, .log.
o Multiple Search Options:
‘ Simple and Advance Search: In Simple Search log analyst can select a file or multiple files or a complete folder and search for both Warning and severe or restrict the search for either of one.
‘ Manual Keyword Based Search: In Keyword Search log analyst can select a file or multiple files or a complete folder and search for the keyword of the Log Analyst’s choice. Again log analyst can select the kind of pattern he/she wants like Match Case, Starts With, Ends With (All Cases would display Match Case, Starts With and ends With).
‘ Automatic search of Standard Java Errors in the log File: When Simple, Keyword or Date and Time range searches are performed on the log files Java Exception are automatically grepped and displayed to the User.
‘ Search based on Date and Time Range: This is a unique feature and more useful feature where a time range can be specified and everything within the time range will get displayed to user
o Multiple Options to Display Errors: Errors which are displayed to the user have different colour Example: – SEVERE ‘ Red and WARNING-Green For the number of files selected corresponding tabbed panes gets generated and displays the errors specific to the file names.
o Close Options: The Tabs created can be independently closed or on right click there is option available to close the entire tabs which are opened
o Provide Highlighters to the Errors found: For all the searches performed Simple, Keyword and Date and Time Range Search Colour Highlighters are provide for the Log Analyst to locate the errors in the file. Different colours indicating severity level.
‘ Simple Search:
SEVERE ‘Red, WARNING-Green
‘ Keyword Search:
Searched Keyword will get a Yellow Highlight.
‘ Date and Time Range search:
Start Time- Green, End Time ‘Green.
o Panels in the UI should be Adjustable: This feature is mainly for the Log Analyst’s readability.
o Display the Progress of the Search: When the User is performing a search notify if it is progressing and when the search is completed.
o Abort Operation: If the user is not interested in performing the search heshe must be able to abort the search at any point of time.
o Open Files: The Tool also enables the Log Analyst’s to open the selected file from the tool with the supporting software required to open that particular file. The files are stored on a table for each of the search options. It is stored along with the path for the Log Analyst to identify which log file it is while performing search on multiple files or a folder.
2.5 NETBEANS- IDE 8.0.1 TOOL
NetBeans is a multi-language software development environment comprising an integrated development environment (IDE) and an extensible plug-in system. It is written primarily in Java and can be used to develop applications in Java and, by means of the various plug-ins, in other languages as well, including C, C++, COBOL, Python, Perl, PHP, and others. NetBeans employs plug-ins in order to provide all of its functionality on top of (and including) the runtime system, in contrast to some other applications where functionality is typically hard coded.
The NetBeans SDK includes the NetBeans java development tools (JDT), offering an IDE with a built-in incremental Java compiler and a full model of the Java source files. This allows for advanced refactoring techniques and code analysis. The IDE also makes use of a workspace, in this case a set of metadata over a flat file space allowing external file modifications as long as the corresponding workspace “resource” is refreshed afterwards. For implementation purpose Java is chosen as the programming language because of few reasons like Platform Independent, object oriented, rich standard library, swing support, applet interface, distributed etc.
CHAPTER 3
SYSTEM REQUIREMENT SPECIFICATION
Software requirement Specification is a fundamental or a base document, which forms the foundation of the software development process. It plays a vital role during the development process. It not only lists the requirements of a system but also has a description of its major feature. An SRS is basically an organization’s understanding (in writing) of a customer or potential client’s system requirements and dependencies at a particular point in time prior to any actual design or development work. The SRS also functions as a blueprint for completing a project with as little cost growth as possible. The SRS is often referred to as the “parent” document because all subsequent project management documents, such as design specifications, statements of work, software architecture specifications, testing and validation plans, and documentation plans, are related to it. It is important to note that an SRS contains functional and non functional requirements only; it doesn’t offer design suggestions, possible solutions to technology or business issues, or any other information other than what the development team understands the customer’s system requirements to be.
3.1 FUNCTIONAL REQUIREMENT
Functional Requirement defines a function of a software system and how the system must behave when presented with specific inputs or conditions. These may include calculations, data manipulation and processing and other specific functionality. In this system following are the functional requirements:-
o Select Search Options (Simple, Keyword or Date and Time Range).
o Select file(s) or Folder.
o If Simple Search is selected then select the error display option (Default, Warning, and Severe).
o If Keyword Search is selected then specify a valid keyword and choose the pattern (Match Case, Starts With, Ends With).
o If Date and Time Range Search is selected then user should type the start and end time to perform the search.
o And then Click on Start search Button.
3.2 NON-FUNCTIONAL REQUIREMENT
Non-functional requirements are the requirements which are not directly concerned with the specific function delivered by the system. They specify the criteria that can be used to judge the operation of a system rather than specific behaviours. They may relate to emergent system properties such as reliability, response time and store occupancy. Non-functional requirements arise through the user needs, because of budget constraints, organizational policies and the need for interoperability with other software and hardware systems or because of external factors.
3.2.1 Availability
All the information of the caller/callee must be available to the user at all the time after installing the application.
3.2.2 Efficiency
This application should consume as much as less resource as possible and should run properly in all circumstances.
3.2.3 Extensibility
This application should be able to get upgrade with changing requirements at various stages.
3.2.4 Platform Compatibility
This application must be compatible with the current platform on which it is built as well as its higher version.
3.2.5 Response Time
This application must have low response time so user can access the information without any delay.
3.2.6 Usability
This application must have an intuitive and user friendly interface for displaying any information.
CHAPTER 4
SYSTEM ANALYSIS
System development method is a process through which a product will get completed or a product gets rid from any problem. Software development process is described as a number of phases, procedures and steps that gives the complete software. It follows series of steps which is used for product progress. The development method followed in this project is waterfall model.
4.1 Model Phases
The waterfall model is a sequential software development process, in which progress is seen as flowing steadily downwards (like a waterfall) through the phases of Requirement initiation, Analysis, Design, Implementation, Testing and maintenance.
Requirement Analysis: This phase is concerned about collection of requirement of the system. This process involves generating document and requirement review.
System Design: Keeping the requirements in mind the system specifications are translated in to a software representation. In this phase the designer emphasizes on:-algorithm, data structure, software architecture etc.
Coding: In this phase programmer starts his coding in order to give a full sketch of product. In other words system specifications are only converted in to machine readable compute code.
Implementation: The implementation phase involves the actual coding or programming of the software. The output of this phase is typically the library, executables, user manuals and additional software documentation.
Testing: In this phase all programs (models) are integrated and tested to ensure that the complete system meets the software requirements. The testing is concerned with verification and validation.
Maintenance: The maintenance phase is the longest phase in which the software is updated to fulfil the changing customer need, adapt to accommodate change in the external environment, correct errors and oversights previously undetected in the testing phase, enhance the efficiency of the software.
4.2 Waterfall Model as Development Method:
o Clear project objectives.
o Stable project requirements.
o Progress of system is measurable.
o Strict sign-off requirements.
o Helps you to be perfect.
o Logic of software development is clearly understood.
o Production of a formal specification
o Better resource allocation.
o Improves quality. The emphasis on requirements and design before writing a single line of code ensures minimal wastage of time and effort and reduces the risk of schedule slippage.
o Less human resources required as once one phase is finished those people can start working on to the next phase.
CHAPTER 5
SYSTEM DESIGN
‘Design’ is defined as ‘The process of applying various techniques and principles for the purpose of defining a process or a system in sufficient detail to permit its physical realization’. Various design features are followed to develop the system. The design specification describes the features of the system, the components or elements of the system and their appearance to end-users.
5.1 Fundamental Design Concepts
A set of fundamental design concepts has evolved over the past three decades. Although the degree of interest in each concept has varied over the years, each has stood the test of time. Each provides the software designer with a foundation from which more sophisticated design methods can be applied. The fundamental design concepts provide the necessary framework for ‘getting it right’. The fundamental design concepts such as abstraction, refinement, modularity, software architecture, control hierarchy, structural partitioning, data structure, software procedure and information hiding are applied in this project to getting it right as per the specification.
5.1.1 Input Design
The input Design is the process of converting the user-oriented inputs in to the computer-based format. The goal of designing input data is to make the automation as easy and free from errors as possible. Providing a good input design for the application easy data input and selection features are adopted. The input design requirements such as user friendliness, consistent format and interactive dialogue for giving the right message and help for the user at right time are also considered for the development of the project. Input design is a part of overall system design which requires very careful attention. Often the collection of input data is the most expensive part of the system, which needs to be route through number of modules .It is the point where the user ready to send the data to the destination machine along with known IP address; if the IP address is unknown then it may prone to error.
5.1.2 Output Design
A quality output is one, which meets the requirements of the end user and presents the information clearly. In any system results of processing are communicated to the users and to other systems through outputs. It is most important and direct source information to the user. Efficient and intelligent output improves the systems relationship with source and destination machine. Outputs from computers are required primarily to get same packet that the user has send instead of corrupted packet and spoofed packets. They are also used to provide to permanent copy of these results for later consultation.
5.1.3 The MVC Design Method
Swing actually makes use of a simplified variant of the MVC design called the model-delegate. This design combines the view and the controller object into a single element that draws the component to the screen and handles GUI events known as the UI delegate. Communication between the model and the UI delegate becomes a two-way street. Each Swing component contains a model and a UI delegate. The model is responsible for maintaining information about the component’s state. The UI delegate is responsible for maintaining information about how to draw the component on the screen. The UI delegate (in conjunction with AWT) reacts to various events that propagate through the component.
The design method that has been followed to design the architecture of the system is MVC design pattern. Swing uses the model-view-controller (MVC) architecture as the fundamental design behind each of its components. Essentially, MVC breaks GUI component into three elements. Each of these elements plays a crucial role in how the component behaves. The MVC design pattern separates a software component into three distinct pieces: a model, a view, and a controller.
o Model: The model is the piece that represents the state and low-level behavior of the component. It manages the state and conducts all transformations on that state. The model has no specific knowledge of either its controllers or its views. It encompasses the state data for each component. There are different models for different types of components. For example, the model of a scrollbar component might contain information about its current position of its adjustable ‘thumb’, its minimum and maximum values, and the thumb’s width. A menu on the other hand, may simply contain a list of the menu items the user can select from. The system itself maintains links between model and views and notifies the views when the model changes state.
Figure 5.1:- Combination of View & Controller into a UI delegate object
o View: The view refers to how you see the component in the screen. It is the piece that manages the visual display of the state represented by the model. Almost all window frames will have a title bar spanning the top of the window. However the title bar may have a close box on the left side or on the right side. These are the examples of different types of views for the same window object. A model can have more than one view, but that is typically not the case in the Swing set.
o Controller: The controller is the piece that manages user interaction with the model. It provides the mechanism by which changes are made to the state of the model. It is the portion of the user interface that dictates how the component interacts with events.
The view cannot render the scrollbar correctly without obtaining information from the model first. In this case the scrollbar will not know where to draw its ‘thumb’ unless it can obtain its current position and width relative to the minimum and maximum. Likewise the view determines if the component is the recipient of user events, such as mouse clicks. The view passes these events on to the controller, which decides how to handle them best. Based on the controller’s decision the values in the model may need to be altered. If the user drags the scrollbar thumb, the controller will react by incrementing the thumb’s position in the model. At that point the whole cycle can repeat.
5.2 Architectural Design
Architectural Design is the conceptual design that defines the structure and behaviour of a system. An architecture description is a formal description of a system, organized in a way that supports reasoning about the structural properties of the system. It defines the system components or building blocks and provides a plan from which products can be procured, and systems developed, that will work together to implement the overall system. The System architecture is shown below.
Figure 5.2:- Log Analyzer System Architecture
Input: Input to the tool is Log file(s), Folder.
Operation: Simple, Keyword and Date and Time Range Search.
Output: Log Analyst expected parsed result.
5.2 Detailed System architecture
The detailed architecture explains how the of flow of the Log Analyzer Tool works Initially when the Log Analyzer Tool is accessed it loads the Initial screen with a tool bar and a display screen. Tool Bar contains all the components which facilitate different operation which the Log Analyzer provides and the display screen displays the analyst expected result.
Figure 5.3:- Detailed Architecture of Log Analyzer
Operations which the tool provides are: –
i. Simple Search.
ii. Keyword Search.
iii. Date and Time Range Search.
5.3 Sequence Diagrams for the System operation
Sequence diagram in Unified Modelling Language (UML) is a kind of interaction
diagram that shows how processes operate with one another and in what order. It is a
construct of a Message Sequence Chart.
5.3.1 Sequence Diagram for Initiating the Tool
Figure 5.4:- Sequence Diagram for Initiating Log Analyzer Tool
This is how the Log Analyzer Tool is initialized. When the user initiates the tool Home screen of the Log Analyzer Tool is displayed to user where the user can further interact with the tool to find out bugs within the log files. User can perform Simple search, Keyword Search and Date and Time Range Search can be performed on the log files to find the bugs in the product further the analyst fixes those bugs using available resources to overcome the bugs in order to keep the product up and running. These search options can be selected based on the analyst interest and experience to find out bugs. If the analyst is not aware of what could be the error in the logs then he/she could perform Simple search along with Error Display options. If the analyst guesses what could be the reason for the failure in the product then he/she could go with Keyword search operation where the Analyst could type the possible error query and check if the failure occurred for the same reason. And If the product was not working only from a particular time then the suggested search operation would be Date and Time and Search.
5.3.2 Sequence Diagram for Simple Search
Figure 5.5:- Sequence Diagram for Simple Search
This sequence diagram shows the interactions between the User and the Simple Search operation
5.3.3 Sequence Diagram for Keyword Search
Figure 5.6:- Sequence Diagram for Keyword Search
This sequence diagram shows the interactions between the User and the Keyword Search operation
5.3.4 Sequence Diagram for Date and Time Range Search
Figure 5.7:- Sequence Diagram for Date and Time Range Search
This sequence diagram shows the interactions between the User and the Date and Time Range search operation.
5.2 Use case Diagram of the system
A use case diagram is a type of behavioral diagram created from a Use-case analysis. Its purpose is to present a graphical overview of the functionality provided by a system in terms of actors, their goals (represented as use cases), and any dependencies between those use cases
.
Figure 5.8:- Use case Diagram for Log Analyzer Tool
The above use case diagram shows the scenario where the user can operate on the Log Analyzer Tool. Firstly Initiate the Log Analyzer Tool, provide options for user to select the search type among the three modules and provide other inputs to the tool to successfully perform search on the Log Files and yield accurate result.
CHAPTER 6
IMPLEMENTATION
The implementation phase of the project is where the detailed design is actually transformed into working code. Aim of the phase is to translate the design into a best possible solution in a suitable programming language. This chapter covers the implementation aspects of the project, giving details of the programming language and development environment used. It also gives an overview of the core modules of the project with their step by step flow.
The implementation stage requires the following tasks.
‘ Careful planning of how the project is to be carried out.
‘ Investigation of system and constraints required to accomplish the project.
‘ Design of methods to successfully abide to future changes.
‘ Evaluation of the changeover method.
‘ Selecting suitable platform to develop the project.
‘ Selecting appropriate language for application development.
6.1 Language Used For Implementation
Implementation phase should perfectly map the design document in a suitable programming language in order to achieve the necessary final and correct product. Often the product contains flaws and gets ruined due to incorrect programming language chosen for implementation. In this project, for implementation purpose Java is chosen as the programming language. Few reasons for which Java is selected as a programming language can be outlined as follows:-
o Platform Independent: Java compilers do not produce native object code for a particular platform but rather ‘byte code’ instructions for the Java Virtual Machine (JVM). Making Java code work on a particular platform is then simply a matter of writing a byte code interpreter to simulate a JVM. What this all means is that the same compiled byte code will run unmodified on any platform that supports Java.
o Java is object-oriented: Java is object-oriented because programming in Java is centred on creating objects, manipulating objects, and making objects work together. This allows creation of modular programs and reusable code.
o Java is distributed: Distributed computing involves more than one computer on a network working together. Java is designed to make distributed computing easy with the networking capability that is inherently integrated into it.
o Java is platform-independent: One of the most significant advantages of Java is its ability to move easily from one computer system to another.
(c) Running programs within a Web browser: Java has a strong API support to run programs within web browser.
o Rich Standard Library: One of Java’s most attractive features is its standard
library. The Java environment includes hundreds of classes and methods in six major functional areas. Language Support classes for advanced language features such as strings, arrays, threads, and exception handling. Utility classes like a random number generator, date and time functions, and container classes.
‘ Input/output classes to read and write data of many types to and from a variety of sources.
‘ Networking classes to allow inter-computer communications over a local network or the Internet.
‘ Abstract Window Toolkit for creating platform-independent GUI
‘ Applet is a class that lets you create Java programs that can be downloaded and run on a client browser.
o Applet Interface. In addition to being able to create stand-alone applications, Java developers can create programs that can download from a web page and run on a client browser.
o Garbage Collection. Java does not require programmers to explicitly free dynamically allocated memory. This makes Java programs easier to write and less prone to memory errors.
o Swing Support. Swing was developed to provide a more sophisticated set of GUI components than the earlier Abstract Window Toolkit. Swing provides a native look and feel that emulates the look and feel of several platforms, and also supports a pluggable look and feel that allows applications to have a look and feel unrelated to the underlying platform.
6.2 Platform Used For Implementation
A platform is a crucial element in software development. A platform might be simply defined as ‘a place to launch software’. In this project, for implementation purpose
NetBeans -IDE 8.0.1 is used.
6.2.1 NETBEANS- IDE 8.0.1
NetBeans is a multi-language software development environment comprising an integrated development environment (IDE) and an extensible plug-in system. It is written primarily in Java and can be used to develop applications in Java and, by means of the various plug-ins, in other languages as well, including C, C++, COBOL, Python, Perl, PHP, and others. NetBeans employs plug-ins in order to provide all of its functionality on top of (and including) the runtime system, in contrast to some other applications where functionality is typically hard coded.
The NetBeans SDK includes the NetBeans java development tools (JDT), offering an IDE with a built-in incremental Java compiler and a full model of the Java source files. This allows for advanced refactoring techniques and code analysis. The IDE also makes use of a workspace, in this case a set of metadata over a flat file space allowing external file modifications as long as the corresponding workspace “resource” is refreshed afterwards.
Swing: The Java Foundation Classes (JFC) consists of five major parts: AWT, Swing, and Accessibility, Java 2D, and Drag and Drop. Java 2D has become an integral part of AWT, Swing is built on top of AWT, and Accessibility support is built into Swing. The five parts of JFC are certainly not mutually exclusive, and Swing is expected to merge more deeply with AWT in future versions of Java. Swing is a set of classes that provides more powerful and flexible components than are possible with the AWT. In addition to the familiar components, Swing supplies tabbed panes, scroll panes, trees, and tables. It provides a single API capable of supporting multiple look-and feels so that developers and end-users are not locked into a single platform’s look-and-feel. The Swing library makes heavy use of the MVC software design pattern, which conceptually decouples the data being viewed from the user interface controls through which it is viewed. Swing actually makes use of a simplified variant of the MVC design called the model-delegate
Swing possesses several traits such as’
o Platform-independence
o Extensibility
o Component-oriented
o Customizable
o Configurable
o Look and feel.
Platform independence both in terms of its expression and its implementation, extensibility which allows for the “plugging” of various custom implementations of specified framework interfaces Users can provide their own custom implementation of these components to override the default implementations. Component-orientation allows responding to a well-known set of commands specific to the component. Specifically, Swing components are Java Beans components, compliant with the Java Beans Component Architecture specifications. Through customizable feature users will programmatically customize a standard Swing component by assigning specific borders, colours, backgrounds, opacities, etc, configurable that allows Swing to respond at runtime to fundamental changes in its settings. Finally look and feel allows one to specialize the look and feel of widgets, by modifying the default via runtime parameters deriving from an existing one, by creating one from scratch, or, beginning with J2SE 5.0, by using the Look and Feel which is configured with an XML property file.
6.3 Modules
This project mainly consists of 3 Modules. They are: –
o Simple Search
o Keyword Search
o Date and Time Range Search
6.3.1 Simple Search Module
Figure 6.1:- Block Diagram for Simple Search
Initially the user is given option to select a single log file or multiple log files or an entire log folder based on the users choice to perform the simple search operation. Then user must select files for which log parsing must be performed .After the User selects the files, the selected file names along with entire path are displayed on a table which allows the analyst to know on what component the search is performed, the tool also allows analyst to open the file within framework to cross check if the analyst is performing the operation on the file which is of interest. It also provides options to delete single file from the table or the entire list of files from the tables. Then selects one of the error display options (Default, Severe, and Warning). By selecting Default the tool displays both Severe and Warning with highlighters for user to easily locate them, on selecting Warning or Severe it displays only them. Further the analyst clicks on Start Search button which performs the search on the files for the selected error display option , displaying the grepped result by creating individual tabs for the files to avoid confusion to know which result belongs to which file. Once the user has started the search, options are given to abort the operation. Every tab changes colour to indicate that the operation has completed in that File. Progress Bar is provided to indicate the operation is still in progress and also a text saying ‘Search Completed’ pops after the entire search has completed.
6.3.2 Keyword Search Module
Keyword Search is used when the analyst already guesses what could be the reason for the failure of the product. So this module allows the analyst to selects the Log files or entire folder of log files, then the analyst enters the possible query for which they think the failure has occurred. Selects the Matching Type (Match Case, Starts With, Ends With, All Cases). Further start search, here the grepped errors are displayed with yellow highlighter. At any point of time the search could be aborted and resumed providing flexibility for the analyst.
6.3.3 Date and Time Range Search Module
Similar to Simple and Keyword Searches Date and Time Range Search greps the content within the Start and End Time and displays it to the user/Analyst to study about the product with information available from the log files . Analyst studies the log information for which the failure occurred in the product and provides relevant solution to overcome the problem faced by the clients
CHAPTER 7
TESTING
System testing is actually a series of different tests whose primary purpose is to fully exercise the computer-based system. Although each test has a different purpose, all work to verify that all the system elements have been properly integrated and perform allocated functions .The testing process is actually carried out to make sure that the product exactly does the same thing what is supposed to do. Testing is the final verification and validation activity within the organization itself. In the testing stage following goals are tried to achieve:-
o To affirm the quality of the project.
o To find and eliminate any residual errors from previous stages.
o To validate the software as a solution to the original problem.
o To provide operational reliability of the system.
During testing the major activities are concentrated on the examination and modification of the source code.
7.1 Unit Testing
Here each module that comprises the overall system is tested individually. Unit testing focuses verification efforts even in the smallest unit of software design in each module. This is also known as ‘Module Testing’. The modules of the system are tested separately. This testing is carried out in the programming style itself. Unit testing exercises specific paths in a module’s control structure to ensure complete coverage and maximum error detection. This test focuses on each module individually, ensuring that it functions properly as a unit. Hence, the naming is Unit Testing. In this step each module is found to work satisfactorily as regard to the expected output from the module. This testing is done to check for the individual block codes for their working. It is done so that when we carry out functional testing then the units which are part of these functionalities should have been tested for working.
7.2 Integration
After successful completion of unit testing or module testing, individual functions are integrated into classes. Again integration of different classes takes into place and finally integration of front-end with back-end occurs.
o Integration of functions into classes
At the start of coding phase only the functions required in different parts of the program are developed. Each of the functions is coded and tested independently. After verification of correctness of the different functions, they are integrated into their respective classes.
o Integration of different classes
Here the different classes are tested independently for their functionality. After verification of correctness of outputs after testing each class, they are integrated together and tested again.
o Integration of front-end with back-end
The front-end of the project is developed in Java Swing environment. The user interface is designed to facilitate the user to input various commands to the system and view the system’s normal and faulty behaviour and its outputs. The back-end code is then integrated with the GUI and tested.
7.3 Integration Testing
Data can be lost across interface. One module can have an adverse effect on another. Sub functions when combined, should not reduce the desired major function. Integration testing is a systematic technique for constructing the program structure. It addresses the issues associated with the dual problems of verification and program construction. The main objective in this testing process is to take unit tested modules and build a program structure that has been dictated by design. After the software has been integrated, a set of high order tests are conducted. All the modules are combined and tested as a whole. Here correction is difficult, because the isolation of errors is complicated by the vast expanse of the entire program.
7.3.1 Top down Integration
This method is an incremental approach to the construction of program structure. Modules are integrated by moving downward, beginning with the main program module. Modules that subordinates to the main program module are incorporated into the structure in either a depth first or breadth first manner.
7.3.2 Bottom-up Integration
This method begins the construction and testing with the modules at the lowest level in the program structure. Since the modules are integrated from bottom to up, processing required for modules subordinate to a given level is always available. Therefore in this case the need for stubs is eliminated. The following integration testing table shows the functions that were combined into different classes and the class as a whole tested for its functionality. This is important to check for error-free interaction between various classes, and maintenance of data integrity.
Table 7.1:- Integration Testing Table
Classes Integrated Functions Integrated in each class Tests Performed Remarks
Class : Main InitComponents()
LoadInitialScreen()
‘ Check whether initial screen with appropriate components displayed to user.
‘ Check if proper Search Panel are opened on selection from the Initial Screen Success
Class: ClosableTabbedPane ClosableTabbedPane() ‘ Check if each Tabbed pane are closable within each search option Editor pane
Class: SimpleSearch DefaultOption()
WarningOption()
SevereOption()
‘ Check for the selected error display if the grepped result is displayed with proper highlighters. Success
Class: SimpleHighlighter
SSHLRedSevere()
SSHLGreenWarning() ‘ Check if displayed result is with proper highlighters Success
Class: SimpleSelectFiles
SSSelectFiles() ‘ Check if File, Files or Folder can be selected Success
Class:SimpleTablePopUp SSTPopUp() ‘ Also Check if files displayed on the Table could be opened within the Framework. Delete single Files and delete all files from the table. Success
Class:SimpleSearchClearPanel SSClearPanel() ‘ Check if on click of the clear button if components of Simple Search is Cleared Success
Class: KeywordSearch MatchCase()
StartsWith()
EndsWith()
AllCase()
‘ Check if the grepped result displayed correctly with respect to the pattern match type selected.
Success
Class:KeywordHighlighter KSHLYellowQuery () ‘ Check if the pattern matching the keyword entered is highlighted with yellow colour Success
Class:KeywordSelectFiles KSSelectFiles() ‘ Check if File, Files or Folder can be selected Success
Class:KeywordClearPanel KSClearPanel() ‘ Check if Clear Button clears the components of Keyword Search Panel.
Success
Class:KeywordTablePopUp KSTPopUp() ‘ Also Check if files displayed on the Table could be opened within the Framework. Delete single Files and delete all files from the table. Success
Class: DATRSearch StartAndEndTime()
‘ Check if the displayed result is with respect to Start and end time specified. Success
Class: DATRSelectFiles DATRSelectFiles() ‘ Check if File, Files or Folder can be selected Success
Class: DATRHighlighter DATRHLGreen() ‘ Check if the Start and End time Highlighted with green colour Success
Class: DATRClearPanel DATRClearPanel() ‘ Check if Clear Button clears the components of DATR Search Panel.
Success
Class: DATRTablePopUP DATRTPopUp() ‘ Also Check if files displayed on the Table could be opened within the Framework. Delete single Files and delete all files from the table. Success
7.4 SYSTEM TESTING
In this stage the software is tested from all possible dimensions for all intended purposes and platforms. In this stage Black box testing technique is normally used. System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system’s compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic. As a rule, system testing takes, as its input, all of the “integrated” software components that have successfully passed integration testing and also the software system itself integrated with any applicable hardware system(s). The purpose of integration testing is to detect any inconsistencies between the software units that are integrated together (called assemblages) or between any of the assemblages and the hardware. System testing is a more limited type of testing; it seeks to detect defects both within the “inter-assemblages” and also within the system as a whole.
7.5 White Box Testing
White box testing (clear box testing, glass box testing, and transparent box testing or structural testing) uses an internal perspective of the system to design test cases based on internal structure. It requires programming skills to identify all paths through the software. The tester chooses test case inputs to exercise paths through the code and determines the appropriate outputs.
While white box testing is applicable at the unit, integration and system levels of the software testing process, it is typically applied to the unit. While it normally tests paths within a unit, it can also test paths between units during integration, and between subsystems during a system level test. Though this method of test design can uncover an overwhelming number of test cases, it might not detect unimplemented parts of the specification or missing requirements, but one can be sure that all paths through the test object are executed. Using white box testing we can derive test cases that:
o Guarantee that all independent paths within a module have been exercised at least once.
o Exercise all logical decisions on their true and false sides.
o Execute all loops at their boundaries and within their operational bounds.
o Execute internal data structure to assure their validity
7.6 Black Box Testing
Black box testing focuses on the functional requirements of the software. It is also known as functional testing. It is a software testing technique whereby the internal workings of the item being tested are not known by the tester. For example, in a black box test on software design the tester only knows the inputs and what the expected outcomes should be and not how the program arrives at those outputs.
The tester does not ever examine the programming code and does not need any further knowledge of the program other than its specifications. It enables us to derive sets of input conditions that will fully exercise all functional requirements for a program. Black box testing is an alternative to white box technique. Rather it is a complementary approach that is likely to uncover a different class of errors in the following categories:-
o Incorrect or missing function.
o Interface errors.
o Performance errors.
o Initialization and termination errors.
o Errors in objects.
Advantages
o The test is unbiased as the designer and the tester are independent of each other.
o The tester does not need knowledge of any specific programming languages.
o The test is done from the point of view of the user, not the designer.
o Test cases can be designed as soon as the specifications are complete.
7.7 Preparation Of Test Data
Preparation of test data plays a vital role in the system testing. After preparing the test data, the system under study is tested using that test data. While testing the system by using test data, errors are again uncovered and corrected by using above testing steps and corrections are also noted for future use.
7.7.1 Using Live Test Data
Live test data are those that are actually extracted from organization files. After a system is partially constructed, programmers or analysts often ask users to suggest data for test from their normal activities. Then, the systems person uses this data as a way to partially test the system. In other instances, programmers or analysts extract a set of live data from the files that they have entered themselves.
It is difficult to obtain live data in sufficient amounts to conduct extensive testing and although the realistic data that will show how the system will perform for the typical processing requirement. Assuming that the live data entered are in fact typical; such data generally will not test all combinations or formats that can enter the system. This bias toward typical values then does not provide a true system test and in fact ignores the cases most likely to cause system failure.
7.7.2 Using Artificial Test Data
Artificial test data are created solely for test purposes, since they can be generated to test all combinations of formats and values. In other words, the artificial data, which can quickly be prepared by a data generating utility program in the information systems department, make possible the testing of all login and control paths through the program.
The most effective test programs use artificial test data generated by persons other than those who wrote the programs. Often, an independent team of testers formulates a testing plan, using the systems specifications.
7.8 Validation of components
Functionality to be tested Input Tests done Remarks
Initial Screen User Interaction with the Screen to Select search Options Check if appropriate panel open with respect to the Search option selected Success
Simple Search panel No files selected or/and No display error options selected Prompt the user that files or/and error display options must be selected in order to perform search Success
Keyword Search Panel No files or/and No Query entered Prompt the user to select files or/and enter a keyword to perform search Success
Date and Time Range Search No Start Time or/and No End Time Entered Prompt the user to enter Start Time or/and End Time Success
CHAPTER 8
INTERPRETATION OF RESULTS
8.1 Snapshots
The following snapshots define the results or outputs that we will get after step by step execution of all the modules of the system.
8.1.1 Log Analyzer Build
The above snapshot shows the time taken to successfully build the Log Analyzer Tool. NetBeans took only 2second to launch the Log Analyzer.
8.1.2 LOG ANALYZER Initial Screen
The above snapshot shows the initial screen after the tool has launched. The Initial Screen has a Search Tab which contains different search options provided to the user with the help of the combo box
8.1.3 Simple Search Initial Screen
The above snapshot is for the Simple Search Option Selected from Different search option provided in the combo box. The Simple Search tab contains a tool bar which holds the following components Select Files Button which opens a file chooser to select files to perform the simple search, a combo box from which the error options can be chosen (Default, Warning and Severe). It also contains a Selected Files table which displays the selected files full path name, the editor pane opens new tabs for the files present on the selected files tables and these tabs are closable, clear button is used to clear the components of the simple search screen.
8.1.4 Simple Search Select Files
The above snapshot shows how files are selected by the file chooser to perform search.
8.1.5 Simple Search Selected Files Display
The above snapshot shows how the path name of the files selected are displayed on the Selected Table.
8.1.6 Simple Search Error Display Option (Default) In Progress
The above snapshot displays that the search is in progress, the error display option selected here is Default. When the search is still in progress the screen where the bugs grepped are displayed will have a white background and a progress bar is used to notify the user that the search is still progressing.
8.1.7 Simple Search Error Display Option (Default) Completed with Highlighters
The above snapshot shows that the search is completed for files selected and for the error display option. The user gets to know if the search is completed for that particular tab when the background colour of the editor pane changes to grey from white and also to improve the readability highlighters are provided to highlight the error display option selected Warning is highlighted with Green and Severe which should be taken more care of is highlighted in Red. Once after the search is complete for all the files selected it is also indicated to the user with a popup text Search Complete.
8.1.8 Simple Search Error Display Option (Warning) In progress
`The above snapshot is shows the output for the Warning error display option type which is progress, where the grepped result is highlighted with green.
8.1.9 Simple Search Error Display Option (Warning) Completed
`The above snapshot is shows the output for the Warning error display option type , where the grepped result is highlighted with green and search is completed.
8.1.10 Simple Search Error Display Option (Severe) Completed
The above snapshot displays the output for the simple search where the error display option selected is Severe which is highlighted with red colour when the search is completed.
8.1.11 Simple Search Error Display Option (Severe) NO RESULT
This snapshot displays the result when the file does not contain Severe error. It indicates the user by displaying the Results Found is zero.
8.1.12 Simple Search Validation
The above snapshot shows the validation for the simple search where a message pops up when the search button is clicked without selecting the files to perform search.
8.1.13 Keyword Search Initial Screen
This snapshot displays the Initial screen of the Keyword search
8.1.14 Keyword Search Select Files
The above snapshot shows how files are selected by the file chooser to perform search. . It has a button for selecting the files to perform search, the keyword text field is provided for the user to enter the keyword to be searched, a combo box is provided to select the matching type (All Cases, Match Type, Starts With, Ends With) and a button to start search. The Keyword search screen also contains the selected files table and a editor pane which opens different tabs for the files selected to perform search. The tabs are created with respect to the files selected and these tabs could be closed individually and also a popup menu is provided to close all the tabs at once. When the search is in progress at any point of time the search could be aborted at resumed.
8.1.15 Keyword Search Open File from selected the Table
The above snapshot shows a log file from the selected table is opened within the framework. Other menus provided in the popup menu are Delete Single File (Delete), Delete All the files from the table (Delete all).
8.1.16 Keyword Search Pattern Match Type- All Cases
The above snapshot displays the result for the Pattern match type- All cases where it displays the result irrespective of the alphabet case sensitivity.
8.1.17 Keyword Search Pattern Match Type- Match Case
The above snapshot displays the result for the Pattern match type- Match Case where it displays the result only for those characters matching the text entered in the keyword text field.
8.1.18 Keyword Search Pattern Match Type- Starts With (Result)
The above snapshot is the output displayed for the pattern matching type- Starts With.
8.1.19 Keyword Search Pattern Match Type- Starts With (No Result)
The above snapshot is the output displayed for the pattern matching type- Starts With. But here since there are no results which Starts With the keyword entered in the keyword text field the result shown to the user is Result found is zero.
8.1.20 Keyword Search Pattern Match Type- Ends With (Result)
The above snapshot is the output displayed for the pattern matching type- Ends With. The result is displayed for Ends With the keyword entered in the keyword text field the result shown to the user with yellow highlighter.
8.1.21Keyword Search Pattern Match Type-Ends With (No Result)
The above snapshot is the output displayed for the pattern matching type- Ends With. The result is displayed for Ends With the keyword entered in the keyword text field the result shown to the user is Results Found is zero since there are no such bug within the file which matches the keyword entered in the keyword text field.
8.1.22 Keyword Search Select Files Validation
The above snapshot is the output for the validation where start search button only after entering the keyword without selecting the files to perform the search.
8.1.23 Keyword Search Query Field Validation
The above snapshot is the output for the validation where start search button is clicked only after selecting the files to perform search without entering the keyword.
8.1.24 DATR Initial Screen
The above snapshot is the screen which is displaying the Initial Screen for the Date And Time Range Search which has Start and End Time Field.
8.1.25 DATR Start Time
The above snapshot shows the result where the Start Time is highlighted with green colour for the grepped result.
8.1.26 DATR End Time
The above snapshot shows the result where the End Time is highlighted with green colour for the grepped result.
8.1.27 DATR Select Files Validation
The above snapshot is a validation output shown for the Date And Time Range where only the time field is entered without selecting the files to perform search.
8.1.28 DATR Time Field Validation
The above snapshot is a validation output shown for the Date And Time Range where only files are selected to perform search without entering the Start and End Tie Fields.
CONCLUSION
This Tool reduces the time of the Analyst in finding the bugs within the Log Files manually. It facilitates the Analyst in finding the bugs within the Log Files or the folder consisting log files in less time providing operations like Simple Search, keyword Search, Date and Time Range Search within the same framework. The selection of the search type depends on the situation, experience of the Analyst. When loading a single Log File onto the application was a problem Log Analyzer searches for errors/bugs in the log file or log files or a complete folder consisting of log files providing proper Highlighters for the Search Type.
FUTURE ENHANCEMENT
In future we could Add Remote host within the frame work which would directly search the bugs for the product on the client machine rather than asking the clients to make a copy of the Log Files in a shared location which the analyst and the client has access to. Another enhancement could be, provide an option to Display Errors like All errors or distinct errors where All errors option could be used when analyst is interested to know the count of the bugs within the log file and Distinct error option could be used to see only one occurrence of the a error which is the cause for the failure.