Sodhanai

Waterfall Vs Agile

For a long time, I have been reading about both Waterfall and Agile methodologies and understand the differences between them. Here is what I could infer and concur -

Documentation @ Waterfall Model – It is comprehensive, detailed and good enough for seeking formal approvals from all the relevant stakeholders of the project as defined in the Project Org Structure

Documentation @ Agile Model – It is short & crisp, detailed enough and good enough for seeking formal approvals from all the relevant stakeholders of the project as defined in the Project Org Structure

What’s the difference here? Waterfall model has been documenting at a very detailed level which is not warranted. Waterfall model tries to be overcautious with the following features -

a) needs a lot of time for maintaining the documentation to ensure that they are up to date and relevant.
b) engages too many stakeholders for formal approvals & sign-offs.

Main reasons why waterfall model insisted on comprehensive documentation are

a) It runs for a considerable long period of time and to ensure that the project (human) resources shouldnt miss out on addressing any key requirements. Remember – Humans are prone for forgetting things!!!
b) Resources (Project Stakeholders) move on before the project gets completed – say with a notice period of one to three months in majority of the cases which is considerably less than the project duration

The solutions for these issues can be -

a) Reduce the time for waterfall cycles
b) Document only the bare minimum required details
c) Limit the approvers and sign-off authorities

Can we still call it as Waterfall Model. Ideally, yes. We can call it as waterfall model for the following basic facts/ characteristics -

a) Design can only be be produced after the requirements are created
b) Softeare Code can only be created after the designs are created
c) Testing can only be performed after the code is ready and is deployed in to the test environment

Even if we call this new solution as Agile, the above 3 characteristics will not change. Fine, what are the benefits of having this Jargon – Agile?

a) Insist to produce only the bare minimum level of documentation, all in the name of “Agile”
b) Resist to produce more comprehensive detailed level of documentation, all in the name of “Agile”

Apart from these benefits/ differences on handling communication around the project stakeholders, at the granular level, there are not much differences between Agile and Waterfall models at all. Remember, the flow is still the same: Requirements -> Design -> Build -> Test.

What a misleading term is ‘Risk Based Testing’?

Once upon a time, long long ago so long ago, I was working with a Test Assurance Manager for drafting the test strategy of a large programme. After reading through the risk based testing approach advocated, explained and advised by the top consultants & professors, I went ahead and explained the same approach to my Test Assurance Manager to adopt this for our large test programme. I was listing down the different testing types that we need to do such as functional testing, performance testing, security testing, etc., and explained to include risk based testing on top of these testing types to prioritise the test items as advocated by these top consultants and professors. He listened to me very patiently and when I finished, he started talking -

The very first thought of “software testing” emerged from the risk of not validating the software against the requirements. Provided this origin, “software testing” itself can be considered as the answer to the risks that any software development projects might possess.

This served as the principle for our further discussions around the test strategy for our test programme. We talked about the list of functions that needs to be tested and we analysed/ forecasted the situation when those functions are not tested before moving the respective software code into production. Based on that we planned the test phases and provided directions in our test strategy for the test team leads to carry out their respective test planning & designing.

From this, I could clearly infer that testing as a discipline revolves around the risks of not doing it as the base and there cannot be anything above testing which is “risk based testing”. If at all, if it is there, it is just nothing but an attempt to prioritise the test items only. Personally, I prefer to call it as ”test item prioritisation” in accordance with the pure literal meaning and I dont appreciate the idea of giving a name like “Risk Based Testing” which is totally misleading. I sincerely thank the Test Assurance Manager for bringing this clarity around the terms “testing” and “risks”.

Are you playing word games with testing terms?

After a long period of time I spent towards learning software testing, I want to share my experience here. I wanted to become a master of software testing and wanted to know each and every nook and corner of the methodologies and techniques associated with software testing. During the course of this exercise, I have gone through lots of theories/ terminologies and tried to apply them in practice. The more I did go through the theories/ terminologies, the more I got confused… For example, a theory explained that the testing life cycle have unit testing, system testing, system integration testing, and user acceptance testing as the different phases of testing. But in practice, the testing life cycle differs from project to project => some projects might include module testing, operational acceptance testing on top of the phases mentioned earlier and some projects might not have system integration testing only in their life cycle.

What does this mean? We can conclude saying that whatever we have learnt about the testing life cycle from the theories is not useful. Is this fair and can we stop going through the theories? Definitely not for the reason being, the act of analysing what is the right testing life cycle requires certain understanding of the theories. So, theories are useful and it has to be tailored for your practical needs.

In another instance, I have gone through terms like Java Testing, Website Testing, Module Testing, User Testing, Sanity Testing, Pipe Cleaning Testing, Shakedown Testing, etc., etc., with all different meaning and perceptions at different situations/ locations. There is no standard definition for these terms and there are various certification/ governing bodies in this world that tries to standardise these terms. These standardised definitions again are not convincing for all the practical scenarios and many a times, I have seen word games popping up between the testers who are on the ground.

But there is one thing in the end that is always convincing in all the practical scenarios… “Testing” as a term with the definition, “validating the application/ feature/ functionality under test against its explicitly and implicitly stated requirements”. You don’t have to formally define & standardise the terms like module testing (testing the module against the explicitly and implicitly stated requirements for that respective module) & website testing (testing the website against the explicitly and implicitly stated requirements for that respective website), as their meaning can be extended with the above definition of the term “Testing” itself.

Can we apply the same logic to “exploratory testing” term and define it? If we use the same logic, we will conclude with a bizarre definition like – testing the exploratory nature of the application under test is called as exploratory testing. Let us think a little bit different here. You will find lots and lots of materials and explanations for this term in the internet. But, in my perspective, this is nothing but testing the application without having the explicit requirements stated in a piece of paper. So what happened to the implicit requirements? Sorry, that will never get mentioned at all for all the projects under all circumstances and hence the name ‘implicit requirements’. How do we test the application under test without the requirements in place? In the first instance, we can apply the above definition of the term ‘testing’ and conclude that we can test an application under test only if the requirements are listed down in a piece of paper & provided to the tester and vice versa. But at any point in time, there cannot be an application built without the requirements and need. The requirements might not be available in a piece of paper but it could be lying in the minds of the business analysts, designers and developers. It is tester’s job to find out what those requirements are – don’t mind eating the brains of your development & business counterparts (please note, it could be anybody who has vested interest in the application under test who knows the requirements, might not be business and development team members alone). Testing the application under test against those identified requirements by eating the brains of your development & business counterparts is known as exploratory testing.

So, in a nutshell, what does this mean? Do not get bogged down by terms and words. Just carry on with your testing against your stated (explicit) & non-stated (implicit) requirements. Mind the theories but apply your minds before applying them into your practical scenarios. Remember, if you draw a tomato on a piece of paper, it cannot be used for cooking. There is no point in playing word games with testing terms and applying the theories in paper straight into your practical endeavours irrespective of their feasibility. Just remember the essence of those terms and carry on with your testing. Happy Testing!!! If you get into any playing any word games, please share your experience below.

Are you doing your performance Testing with Right Execution Process

My fellow blogger Sanky was keen to point the right attitudes for testing, I would like to add up to ensure that the right attitude should align in the right way of test execution to achieve the required success.

There are three types of performance test cases that are typically executed: response time, stress, and reliability testing. It is important to differentiate between the three since they are intended to measure different KPIs (key performance indicators).

Specialized members of the testing and system administration organizations, who have ownership of the system architecture and infrastructure, typically manage performance tests.

Execute Test

Execute each script for a single user to validate the health of the environment. A low user-load baseline should be obtained before attempting the target user load. This baseline allows you to measure system scalability by comparing results between the baseline and target loads.

Users need to be started at a controlled rate to prevent excessive resource utilization due to large numbers of simultaneous logins. This rate depends on the total configured capacity of the system. For every 1000 users of configured system capacity, you should add one user every three seconds. For example, if the system is configured for 5000 users, you add five users every three seconds.

Excessive login rate causes the application server tier to consume 100% CPU, and logins begin to fail. Think times should be randomized during load testing to prevent inaccuracies due to simulated users executing transactions simultaneously. Randomization ranges should be set based on determining the relative think times of expert and new users when compared to the average think times in the script.

Performing SQL Trace

Since poorly formed SQL or sub optimal database tuning causes many performance issues, the first step to improve performance is to perform SQL trace. SQL trace creates a log file that records the statements generated in the Siebel object manager and that are executed on the database. The time required to execute and fetch on an SQL statement has a significant impact on both the response time seen by end users of a system and on the system’s resource utilization on the database tier. It is important to discover slow SQL statements and root cause, and fixes issues before attempting scalability or load tests, as excessive resource utilization on the database server will invalidate the results of the test or cause it to fail.

To obtain an SQL trace

Set a breakpoint in the script at the end of each action and execute the script for two iterations.
Enable EvtLogLvl (ObjMgrSqlLog=5) to obtain SQL traces for the component on the application server that has this user session or task running.
Continue executing the script for the third iteration and wait for the breakpoint at the end of action.
Turn OFF SQL tracing on the application server (reset it to its original value or 1).
Complete the script execution.

The log file resulting from this procedure has current SQL traces for this business scenario. Typically, any SQL statement over 0.1 seconds is considered suspect and needs to be investigated, either by optimizing the execution of the query (typically by creating an index on the database) or by altering the application to change the query.

Measure System Metrics

Results collection should occur during a measurement period while the system is at a steady state, simulating ongoing operation in the course of a business day. Steady state is achieved once all users are logged in and caches (including simulated client-side caches) are primed. The measurement interval should commence after the last user has logged in and completed the first iteration of the business scenario.

The measurement interval should last at least one hour, during which system statistics should be collected across all server tiers. We recommend that you measure the following statistics:

CPU
Memory
System Calls
Context Switches
Paging rates
I/O waits (on the database server)
Transaction response times as reported by load testing tool

NOTE: Response times will be shorter than true end-user response times due to additional processing on the client, which is not included in the measured time.

The analysis of the statistics starts by identifying transactions with unacceptable response times, and then correlating them to observed numbers for CPU, memory, I/O, and so on. This analysis provides insight into the performance bottleneck.

Monitor Failed Transactions

Less than 1% of transactions should fail during the measurement interval. A failure rate greater than 1% indicates a problem with the scripts or the environment.

Typically, transactions fail for one of the following three reasons:

Timeout. A transaction may fail after waiting for a response for a specified timeout interval. This can be caused by a resource issue at a server tier or by a long-running query or script in the application.

If a long-running query or script is applicable to all users of a business scenario, it should be caught in the SQL tracing step. If SQL tracing has been performed and the problem is only seen during loaded testing, it is often caused by data specific to a particular user or item in the test database. For example, a calendar view might be slow for a particular user because prior load testing might have created thousands of activities for that user on a specific day. This would only show as a slow query and a failed transaction during load testing when that user picks that day as part of their usage scenario.

Long-running transactions under load can also be caused by consumption of all available resources on some server tier. In this case, transaction response times typically stay reasonable until utilization of the critical resource closely approaches 100%. Resource utilization across the server tiers should be closely monitored during testing, primarily for data gathering purposes, but also for diagnosing the resource consumption problem.

Very often, a long-running query or script can cause consumption of all available resources at the database server or application server tier, which then causes response times to increase and transactions to time out. While a timeout problem may initially appear to be resource starvation, it is possible that the root cause of the starvation is a long-running query or script.

Improving the Testing Process

This section describes the steps you can take to make iterative improvements to the testing process.

After the initial deployment, regular configuration changes are delivered in new releases. In addition, Siebel Systems delivers regular maintenance and major software releases. Configuration changes and new software releases must be tested to verify that the quality of the system is sustained. This is a continuous effort using a phased deployment.

Hope this would be more useful to performance test leads & technical business leads to ensure they are adhering to the right execution process there by they are in right success path.

Sodhanai

Recent Posts

Archives

Categories

Meta