@Test\nvoid givenHungryThenCook10BreadsWithMyWife() throws InterruptedException {\n var basket = new ConcurrentLinkedQueue<>();\n var knife = new Knife();\n var startGate = new CountDownLatch(1);\n var endGate = new CountDownLatch(10);\n\n Function<String, Runnable> preparation = topping -> () -> {\n try {\n startGate.await();\n } catch (InterruptedException e) {\n throw new RuntimeException(e);\n }\n synchronized (knife) {\n var bread = new Bread();\n knife.setTopping(topping).applyToppingTo(bread);\n basket.add(bread);\n endGate.countDown();\n }\n };\n\n // (...) omitted for brevity\n}Now the fix of our code is just a matter of adding a synchronized block using the knife as the lock. With this, only one thread at a time can take the knife and execute the code inside it.
\nThread blocking
\n\"Oh! This is the solution for everything, we should synchronize everything!\" Think twice. Synchronization isn't for free; it creates a performance problem, as now all other threads are blocked waiting for the availability of the lock, which in our case is the knife.
\nIn our kitchen, we could say that when my wife touches the knife I become blocked, waiting for her. If the phone rings, she should say: \"darling, please answer the phone\". But I will be blocked, I'll not be able to do anything except wait for her to leave the knife.
\nA blocked thread goes to the TIMED_WAITING state, and during this time it can not be used. It is really advisable to keep track of how many threads our applications are using because threads are a scarce resource.
\nBut there is light at the end of the tunnel, at least in this concern you should know that Java 19 is coming up with the concept of Virtual threads, in its Project Loom. This new feature will cut the one-to-one relationship between Java threads and OS threads, which means that more than one Java virtual thread will be able to use the same OS thread. With that, when some virtual thread gets blocked by an I/O operation, for example, another virtual thread can perform calculations on that same OS thread.
\nBut, as we said, until there we should use our threads with wisdom.
\nNonblocking Algorithms
\n\"But, what to do?\", you could ask. Well, if you need better performance you can use Nonblocking Synchronization. Nonblocking algorithms are widely used by JVMs implementations, Operational Systems, and in Java itself to implement locks and other concurrent data structures.
\nThose Nonblocking algorithms are way more complicated to design and implement, but they provide better performance by avoiding the thread block itself and give us other benefits like being immune to Deadlock.
\nTo provide a simple example of Deadlock, imagine that my wife and I are in the living room. The remote controller of the TV and the sofa are Locks. To watch TV, a person needs to sit on the sofa and press the button on the remote controller. A Deadlock would occur if I get the remote controller and my wife sits on the sofa. She will be blocked waiting for the remote controller, and I'll be blocked waiting for the sofa. And we will be like that \"until death does us part.\"
\nLet's go back to the Nonblocking algorithm. First of all, we need a way to ensure atomicity to our tasks. And it is possible because modern computer processors have special instructions for managing access to shared data. A common approach is the implementation of the compare-and-swap (CAS) instruction. It guarantees us that the access to the value and its update will occur as a single task.
\nJava's Concurrent package provides us with some so-called atomic classes. Let's take AtomicInteger as an example, its documentation states that it is \"An int value that may be updated atomically\".
\nThe following code shows us the same algorithm, but only the last one achieves thread-safe by delegating its thread-safety to an AtomicInteger:
\nclass Counter {\n private Integer count = 0;\n\n public void incrementCount() {\n this.count = count + 1;\n }\n\n public int getCount() {\n return count;\n }\n}\n\nclass ThreadSafeCounter {\n private AtomicInteger count = new AtomicInteger();\n\n public void incrementCount() {\n this.count.incrementAndGet();\n }\n\n public int getCount() {\n return count.get();\n }\n}And how does AtomicInteger ensure atomicity? Using the CAS algorithm! Its incrementAndGet method executes a code that could be simplified like this:
\nprivate int value;\n\npublic final int incrementAndGet(int expectedValue) {\n int oldValue;\n do {\n oldValue = value;\n } while (!compareAndSetInt(oldValue, oldValue + 1));\n return oldValue;\n}To start, it gets the current value and stores it in a local variable. Then using the compareAndSetInt method it first checks if, at that precise moment, the current value is the same that was earlier fetched. If it is so, it proceeds with the update and then returns True. If the current value was changed by another thread, the compareAndSetInt method returns False and restarts the process.
\nIt is worth saying that the compareAndSetInt method is a native method, which uses the most efficient instructions provided by the underlying hardware.
\nWith this new knowledge, what about trying it on our bread? Let's remove the blocking synchronization and create the faster bread with butter that a computer could have (and some with hazelnut cream too, for the sake of my wife's happiness). We can do this using an AtomicReference.
\npublic class Knife {\n private final AtomicReference<String> topping = new AtomicReference<>();\n\n public Knife setTopping(String topping) {\n while (!this.topping.compareAndSet(null, topping)) {}\n return this;\n }\n\n public Knife applyToppingTo(Bread bread) {\n String currentTopping;\n if ((currentTopping = topping.getAndUpdate(s -> null)) != null)\n bread.setTopping(currentTopping);\n return this;\n }\n}Of course, we did that after removing the synchronized block from our test. Good, fast, and easy, and without thread blocking! But, please pay close attention to this piece of code. We have some problems there as it is some kind of adaptation of the canonical CAS algorithm.
\nThe hard part of building nonblocking algorithms is to isolate the shared memory into an atomic single change. But in our baker issue, we have a two-step action: first, we set the topping to the knife, and then we add the topping to the bread. So we have two states to keep track of: the knife and the bread.
\nAs we said, this code works and does not block the thread, but doesn't give us all the advantages of the CAS algorithm, because we still can have a Livelock issue.
\nA Livelock issue can occur even when the thread is not blocked, if it is waiting for an operation that will never succeed. In our code, we used the topping value as a kind of lock. The first thread that moves it from null to another value, it will put all other threads waiting for that value to become null again. It works, but if any thread fails to add the topping to the bread, which moves the topping value back to null, all the other threads will be waiting forever.
\nThe control mechanism, when more than one thread fights for the knife topping, is done by retrying again immediately, like in the CAS. But the canonical CAS will not fail indefinitely if any other thread fails. It will just finish its job as soon as no other thread touches the shared memory when it is still working.
\nImmutability
\n\"So, what to do!? I'm reading this to find answers, not questions!\" Ok, I'll let you know. The other side of the coin, in the context of thread safety, is Immutability. If we use an immutable object, virtually all problems with atomicity are gone. That's because all problems that we are facing here are based on threads fighting to access the same shared memory. If the memory that they want to access cannot be modified, those risks are gone and thread safety is achieved.
\nWe can summarise that an Immutable object is one that can't be changed after its creation.
\nJava introduced the Record, which is a simple way to create immutable objects. We just need to provide the type and name of the fields. The public constructor, equals, hashCode, and toString methods, along with the private, final fields, are all given for free to us, by the compiler.
\nTake a look at the following code:
\npublic record Knife (String topping) {\n\n public Knife setTopping(String topping) {\n return new Knife(topping);\n }\n public Knife applyToppingTo(Bread bread) {\n if (topping != null)\n bread.setTopping(topping);\n return new Knife(null);\n }\n}Some could argue that immutable objects are a waste of resources, but it isn't true in the majority of cases. Dealing with immutable objects offers us performance advantages when it frees us from using synchronization or creating defensive copies, it even reduces the impact on generational garbage collection.
\nBut talking about defensive copies, is the record really immutable? Look at this test:
\n@Test\nvoid givenReferenceToRecordThenItShouldBeImmutable() {\n record Parent(String name, List<String> childrenNames){};\n var childrenNames = new ArrayList<>(List.of(\"Haru\"));\n var parent = new Parent(\"Bruno\", childrenNames);\n\n var parentBeforeNewChild = parent.toString();\n childrenNames.add(\"Ayumi\");\n\n assertThat(parent).hasToString(parentBeforeNewChild);\n}This task will fail with the following log:
\njava.lang.AssertionError: \nExpecting actual's toString() to return:\n <"Parent[name=Bruno, childrenNames=[Haru]]">\nbut was:\n <Parent[name=Bruno, childrenNames=[Haru, Ayumi]]>We have to keep in mind that in the way that we created the Parent record we have kept a reference to its internals. A record is still unmodifiable, but to achieve immutability you have to avoid exposing its inner state. In our example, we can avoid that by overriding the constructor and coping the list of children's names:
\nrecord Parent(String name, List<String> childrenNames){\n public Parent(String name, List<String> childrenNames) {\n this.name = name;\n this.childrenNames = new ArrayList<>(childrenNames);\n }\n};Now the test is successful, but we are still publishing the internal state of the record. The getter method for the property childrenNames is publishing the reference to the record's inner state, allowing it to be modified:
\n@Test\nvoid givenReferenceToRecordThenItShouldBeImmutable() {\n record Parent(String name, List<String> childrenNames){\n public Parent(String name, List<String> childrenNames) {\n this.name = name;\n this.childrenNames = new ArrayList<>(childrenNames);\n }\n }\n var childrenNames = new ArrayList<>(List.of(\"Haru\"));\n var parent = new Parent(\"Bruno\", childrenNames);\n\n var parentBeforeNewChild = parent.toString();\n parent.childrenNames().add(\"Ayumi\");\n\n assertThat(parent).hasToString(parentBeforeNewChild);\n}This code will fail as well:
\njava.lang.AssertionError: \nExpecting actual's toString() to return:\n <"Parent[name=Bruno, childrenNames=[Haru]]">\nbut was:\n <Parent[name=Bruno, childrenNames=[Haru, Ayumi]]>To avoid that you have at least two options: override the getter and return a copy of our list, or the one that I prefer: save our internal state as an unmodifiable list:
\nrecord Parent(String name, List<String> childrenNames){\n public Parent(String name, List<String> childrenNames) {\n this.name = name;\n this.childrenNames = List.copyOf(childrenNames);\n }\n}Like this, every time someone tries to update our inner state, they will get a UnsupportedOperationException, which I believe is what we should expect from trying to modify something supposed to be immutable.
\nSummary so far
\nWhat we have seen so far?
\n- \n
- \n
We learned more about Race Conditions, in real life and in the software, and the main solution tool from Java: the monitor pattern, using the synchronized keyword;
\n \n - \n
After understanding the downside of locking, we were introduced to the compare-and-swap (CAS) nonblocking algorithm, which despite being hard to develop can lead us to better performance and protect us from Deadlock
\n \n - \n
For the cases when we are unable to isolate our shared memory, or even in other scenarios where we just need more protection, we talked about immutability and how immutable objects can grant us thread safety.
\n \n - \n
We then revisited the java Record, which is a fairly easy way to create an immutable object, but that is not immune to bad programming technic. To really achieve immutability in our objects we saw how we should avoid publishing its inner state.
\n \n
Conclusion
\nOh, it was an amazing journey, we started with a brief introduction about threads, how to properly test them, and we met some new friends from Java's Concurrency package. With new algorithms and tools under our belt, we are ready to travel to the multithreaded world riding our thread-safe software. I'll now eat my bread with butter, see you soon!
\nAppendix
\nBaking delicious cakes with AssertJ's Condition entity:
\n@Test\nvoid testTheCake() {\n record Cake(String name, boolean hasFruit, int weightInKg){};\n\n var chocolateCake = new Cake(\"Chocolate\", false, 5);\n var strawberryCake = new Cake(\"Strawberry\", true, 3);\n\n var bakery = List.of(chocolateCake, strawberryCake);\n\n var heavyCakes = new Condition<Cake>(cake -> cake.weightInKg >= 3, \"heavy cake\");\n var bigChocolate = new Condition<Cake>(cake ->\n cake.name.contains(\"Chocolate\") &&\n cake.weightInKg >= 5\n , \"big chocolate\");\n\n assertThat(bakery)\n .haveExactly(2, heavyCakes)\n .haveExactly(1, bigChocolate);\n}When a software-based company grows, it creates lots of new code, which means multiple engineers may be facing the same problems across the whole platform. They will usually come up with various solutions to fix it. Some of them might be generic and reusable, while others will be tailored to one’s needs. As it usually happens, the solutions will then be bundled together to create libraries. This is very good, up to a certain point, since over the years, engineers will create them by the dozens and this, in turn, will bring about other hurdles such as:
\n- \n
- how to maintain multiple libraries or prevent buggy code from getting in \n
- how to release one of them or many at the same time \n
- how to ensure a smooth process for updating libs for service maintainers \n
- how to ensure cross-compatibility for all of the libs \n
This was quite a big pain point for AUTO1 software developers as we have 60+ common libraries that need to work perfectly in 600+ production services across multiple business units and products. One simple mistake, for example, a wrong configuration change leftover after a debug session can end up in disaster.
\nEnabling protection against direct commits for main branches will address the first hurdle - only pull requests with double approvals and tested by CI can be merged.
\nThe next 2 issues are more about ease of usability for the developers, as they might be reluctant to upgrade their service if the process is troublesome. So, in the best-case scenario, our end-user should periodically update only a single property. In addition, our common library developers need to be able to easily publish their changes.
\nThe solution to it was the release trains, usually known as BOMs or for AUTO1 developers, service-commons-dependencies. After successful implementation, you will just need to trigger CI build to get the newly released version, which in turn can be pasted into a pom/gradle file. So, right now everything looks good, but can we be 100% sure that our changes are going to work? Perhaps there is a very small bug in a messaging lib or we will lose tracing for incidents. That’s unacceptable. So let’s ensure that will never happen by introducing automatic real use case testing.
\nIn our approach, we thought most of the problems could be revealed by running extensive integration testing. The test harness probably catches all bugs in our codebase and simply fails when needed preventing the release of a buggy version.
\nBut no matter what you do or how extensive your tests are, you are never going to be able to perfectly mock the production environment.
\nEnter the End to End testing of newly released service-commons-dependencies, more commonly known in AUTO1 from here on out, “E2E testing”.
To get it done, we need CI pipelines that will automatically test our new versions of release trains that are triggered either manually or automatically every week. All tests need to be as close to the real-world scenarios as possible. A practical solution is to create a new service that will include all of the most popular use cases of the aforementioned libs. Then we need to deploy it to all of our environments - as previously mentioned there is always something different between local, QA, and production environments, so we need to cover them all. The tests must be the same for all environments. But the trigger may differ - for example, we are using our test environments to prepare new releases for production.
\nThe following diagram illustrates the basic principle of how it should work:
\n![](\"https://lh6.googleusercontent.com/-K5rLPtggaNeQpml856NbA9UyUU6_scnBUgUE_W-RA7MVORz_dYAzrQ_jg7a9vBfQJHqmdosoCH7Evd0dbTF8-kzH0dlygQPeoqQu47__E5Pky0GmeyhlcJ1UWNuQrC7j5s3MZvQhj_EoSgwBA\")
In our implementation, we chose to use Jenkins with JobDSL pipelines and Groovy for scripting. The first stage of the E2E process is checking whether a new version of service-commons-dependencies exists by using the maven versions plugin:
\nmvn -U versions:update-property -Dproperty='service-commons-dependencies.version'To check for any changes:
\ndef hasChanges = sh(returnStatus: true, script: \"git diff --exit-code\")If nothing new was found let’s just abort the pipeline and wait for another CRON or manual trigger.\nIn case a new release was found, we need to build it, deploy it, and run E2E tests.\nIf there are no failures the pipeline will create a new core-test-service release.\nFor those cases, we are using parameterized jobs, for example, to trigger the E2E tests we can use the following commands:
\nbuild(job: 'java/core-svc/core_test_end_to_end_test', parameters: [booleanParam(name: 'SEND_SLACK_NOTIFICATION', value: \"true\"), string(name: 'RELEASE_TRAIN_VERSION', value: service_commons_dependencies_version)])The new release will be picked up by the production pipeline by checking the latest tag creation time in our repository:
\nString latestTagDateString = sh(returnStdout: true, script: \"git rev-list -1 \\$(git describe --abbrev=0 --tags) --pretty=oneline --format=%ci | awk 'NR%2==0'\")\n\ndef proceedWithE2ETesting = wasTagReleasedRecently(latestTagDateString)\n\n@NonCPS\n\n def wasTagReleasedRecently(String lastTagDateString) {\n\n def formatter = DateTimeFormatter.ofPattern(\"yyyy-MM-dd HH:mm:ss Z\")\n\n def lastTagDateTime = OffsetDateTime.parse(lastTagDateString.trim(), formatter)\n\n def currentTimeMinus2hours = OffsetDateTime.now().minus(2, ChronoUnit.HOURS)\n\n return lastTagDateTime.isAfter(currentTimeMinus2hours)\n\n}If the release of core-test-service was created within the last 2 hours we will deploy it by triggering another pipeline. As the last step, it will run E2E tests. As mentioned above, both testing environments will use a shared codebase for acceptance testing. It can be done by setting up one extra job that will fetch an external repository using this block:
\ndefinition {\n cpsScm {\n scm {\n git {\n remote {\n github(\"core-e2e-tests\", 'ssh')\n credentials(\"token\")\n branch(\"master\")\n }\n extensions {\n wipeOutWorkspace()\n }\n }\n }\n\n scriptPath(\"jenkins/core_e2e_tests.gvy\")\n }\n}We chose Cucumber as our main test framework since it will give us unparalleled leverage compared to other choices. Tests are written using natural language in a way that allows for troubleshooting to be done basically at a glance. For example here is the scenario that tests the libraries responsible for sending messages in our platform exposed via HTTP endpoint:
\nScenario: Should have a working API endpoint for creating a message\n\nGiven an object with generated id and message: This is a test\nWhen making a POST call on messaging endpoint with the body containing the given object in JSON representation\nThen should result in a 201 response status code\nAnd should receive an empty bodyThe above, combined with this generated report, is a joy to work with.\n
![](\"https://lh5.googleusercontent.com/qyj0MbKLCEPO9M3Q8kalZoXkgfgcxddnmxk6lYUu0DrbBHhT747lCyy1SFhZDHKZXOGwUqcnsDFDPM2_rWAhnWC5l5m58RVcpsWWrBmwHIZgd3tKxSmhC6aqZHsB016Z7MXAhT0EX7mnT_N3_w\")
Furthermore, we decided to create a dedicated slack channel to notify us about the new releases and test results.
\nSo after getting this message
\nSanity checks for new release train X.Y.Z on env:\n\n * REST API simple entity integration ✅\n * REST API dated entity integration ✅\n * REST API auditable entity integration\n * Messaging integration ✅\n * Messaging FIFO integration ✅\n * Storage integration ✅\n * Health integration ✅\n * Metrics integration ✅\n * Logging integration ✅We can finally get our much-needed stress-free beauty sleep, not that it will help us much, right? Hopefully, this small article will get you in the mood for some nice automation work in your company. Have fun and enjoy your peaceful nights.
","fields":{"slug":"/e2e-testing-internal-libraries/","tags":["auto1","engineering","end2end","testing","internallibraries"]}}},{"node":{"id":"a26825f5-bab1-5ad6-bc95-41f8c672ba4c","frontmatter":{"category":"QA","title":"Efficient Testing - Automate Early","date":"2022-06-01","summary":"Early bird gets the worm - or how preparing in time can reduce the effort needed in manual testing.","thumbnail":null,"authorName":"Joanna Tyka","authorDescription":"Joana is a QA Automation Engineer in our Katowice office.","authorAvatar":{"relativePath":"pages/efficient-testing-2/JoannaT.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAATABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAUCBAH/xAAVAQEBAAAAAAAAAAAAAAAAAAABAv/aAAwDAQACEAMQAAABqc+OOWqwSV4DXFH/xAAcEAEAAQQDAAAAAAAAAAAAAAACAwABBCEQETH/2gAIAQEAAQUCSsDHJd8ZS6gie6yaWk/f/8QAFREBAQAAAAAAAAAAAAAAAAAAASD/2gAIAQMBAT8BI//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EAB0QAAEEAgMAAAAAAAAAAAAAAAEAAhARAyFBUXH/2gAIAQEABj8CLjwtgC4I7TGwPVjqP//EABsQAQEBAAIDAAAAAAAAAAAAAAERABAxIVFx/9oACAEBAAE/IeogXCDAUjcuKkWDGQR7X7rlS/fEZ4Kl4P/aAAwDAQACAAMAAAAQ9AjA/8QAFxEBAQEBAAAAAAAAAAAAAAAAARARMf/aAAgBAwEBPxACOw5P/8QAFxEBAQEBAAAAAAAAAAAAAAAAAAERMf/aAAgBAgEBPxC9Yqv/xAAdEAEAAwADAAMAAAAAAAAAAAABABEhMUFhgaHB/9oACAEBAAE/ELDbyryGiEG8eM0myDEXXZjO3Q44B9sdxnK12+IIWkHfEAGuz9Z//9k=","width":50,"height":50,"src":"/static/d453ba1bcdc86e2db725a048eefd13a6/d2d31/JoannaT.jpeg","srcSet":"/static/d453ba1bcdc86e2db725a048eefd13a6/d2d31/JoannaT.jpeg 1x,\n/static/d453ba1bcdc86e2db725a048eefd13a6/0b804/JoannaT.jpeg 1.5x,\n/static/d453ba1bcdc86e2db725a048eefd13a6/753c3/JoannaT.jpeg 2x,\n/static/d453ba1bcdc86e2db725a048eefd13a6/31ca8/JoannaT.jpeg 3x"}}},"headerImage":null},"html":"Efficient Testing - Automate Early
\nI don't have time for automation - how often do you say that? I said that once and now I feel a bit ashamed thinking about it. To never repeat this mistake I had to find some solution, to not only have enough time for automation but also to be much faster using it. I realized that when I was repeating the same test case manually for the 5th time, it was incredibly boring and each try was taking 15 minutes of my time. Today I would like to share my experience with you on how to make it faster and more pleasant. In the big picture, it is all about being agile, splitting the job between small pieces, done as fast as possible. Before the feature is fully ready for tests we have so many different opportunities to start working on automation.
\nPlanning and reconnaissance
\nAn hour of planning can save you 10 hours of doing - Dale Carnegie
\nAs soon as test scenarios are prepared, you can select candidates for automation and choose which ones should be executed on API level and which on UI. It should be a part of a well-prepared automation strategy (more about test automation strategy here https://auto1.tech/efficient-testing-1/ ). Check whether all necessary API calls are available for you (maybe you need to ask developers to create new endpoints?), what will be the steps to prepare the environment for the test, explore existing automation infrastructure and recognize how many new methods and classes you will need to prepare. Sometimes our features are settled on some other pages or we need to go through other features or applications to do our tests. As these parts already exist, it is a great time to cover it with automation if nobody hasn't done that before. It is also a great time to consider which tools will be needed and find new libraries that will be more effective.
\nTest data preparation
\nTest data means all that needs to be prepared before your real test will start. When you are sure that you know all the preconditions and steps required to arrange the system before beginning the testing, you can recognize how to prepare it just when the first version of the specification is written. For example, let's take an order in the shop. When developers write their code you can write your own which will prepare test data via API, so following our example the order in specified state. To make it more efficient you can store example of the order in the queue (read more about it here https://auto1.tech/faster-automation-tests/). It will be so helpful when you will need to execute the first most basic test and speeds up the whole process significantly. Developers in your team will be also very grateful for the created test data that they can take and quickly test their changes on.
\nPage Object skeletons
\nIf you use Page Object Pattern you can create a page objects skeleton just now based on the design. Prepare all classes storing variables with empty locators, write all methods which will use these locators to interact on the page and prepare bigger steps that will do your manual work. When the front end part of the feature will be implemented you can just fill in locators one by one look at the DOM and adjust steps easily. With unique locators added by developers, it will be almost a pure pleasure.
\n![](\"https://i.postimg.cc/tXm79NqD/image.png\")
Let's take this design of the saved searches page as an example and imagine a test scenario that verifies that saved searches are displayed on the proper page and the user can use a saved search to view the current results of searching. We know that to be able to execute this scenario we need the signed-in user so we can prepare this user just now. Looking at this picture we already know that we have an upper menu that should be already covered with automation (if it isn't covered, the time to do it is just now) and we need to use it to open my account view. We have also a side menu to open the saved searches page from my account context. We know how the saved search will look and which button we will need to click to open search results. Having all this information we can prepare the saved search page class, empty locators for all elements and methods that will interact with the page.
\nDoubts?
\nMany times I saw this approach to automation. I will do everything manually and leave the automation creation at the end or it doesn't make any sense to create automation now, because I will have to change it. Have you seen something similar? But how much time will be needed then to write all automated tests? Two weeks, or more? Who will give this time, when a new urgent feature is upcoming? From my perspective, it is really hard to find that person and it is well-argued. That's why in my humble opinion it is much better to split your time into small sessions before the feature is ready for tests and then have all automation components already done, then just write tests on demand. Isn't it beautiful having valuable tests from the start and be able to use them as regression tests at any time?
\nIn real life
\nI will give you my example: I was testing integration between two systems. Requirements were very agile, five teams were engaged in this feature and multiple layers of integration were created, so we had many bugs by the way. I had to repeat every test scenario 10 times on average. Manually test data preparation for each took 10 minutes, the real testing part between 5 and 10 minutes. After the third attempt, I was tired and frustrated. The more tired and frustrated I was - the more mistakes I made. It was a critical point for me to start writing automated tests. Of course, I had to spend a few hours to have it ready, but what a relief I felt when the 15 minutes scenario was done by my automation in 3 minutes. The other big benefit was that automation doesn't make mistakes and I could run my tests how many times I needed so I've stopped caring about the number of repetitions. I was able to share these tests with my colleague that needed them for testing the other part of this integration. My rough calculations show that I saved a few days and so many hairs which I would have ripped out of my head if I had to do all these tests manually to the end. When I was asked about regression I needed only a few minutes of my effort.
\nConclusion
\nWe have so many opportunities to be well prepared for upcoming features and to reduce the effort needed in manual testing. Creating smart automation will develop our coding skills and help to save time and frustration. I am sure that there are many other ways to receive significant benefits by automating early, we just need to use them.
","fields":{"slug":"/efficient-testing-2/","tags":["auto1","engineering","qa","testing"]}}},{"node":{"id":"43a222cd-fe2c-5cb5-af76-53e7d4ad29ec","frontmatter":{"category":"QA","title":"The journey of a QA engineer in AUTO1 Tech","date":"2022-03-31","summary":"Check out Dhivya's journey in our QA department & how our stack and team contributed.","thumbnail":null,"authorName":"Dhivya Venugopal","authorDescription":"Dhivya is a Junior QA Engineer in the AUTO1 Berlin office.","authorAvatar":{"relativePath":"pages/qa-engineer-journey/dhivya.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAEEBQP/xAAWAQEBAQAAAAAAAAAAAAAAAAABAAL/2gAMAwEAAhADEAAAAXWvULaZ6pqGyQn/xAAcEAACAQUBAAAAAAAAAAAAAAABAgADBBESIRP/2gAIAQEAAQUCpnisScxMquzerXOJXGkTon//xAAVEQEBAAAAAAAAAAAAAAAAAAARIP/aAAgBAwEBPwFj/8QAGBEAAgMAAAAAAAAAAAAAAAAAAAECECH/2gAIAQIBAT8BUTb/AP/EABwQAAEEAwEAAAAAAAAAAAAAAAAQESFBARIxMv/aAAgBAQAGPwIy/KSTY8EDUn//xAAbEAACAwEBAQAAAAAAAAAAAAABEQAxQSFRgf/aAAgBAQABPyHykFbsz0UKMJmkC9nELAfyNrrluBBEa2WJpCJ//9oADAMBAAIAAwAAABCHMMH/xAAZEQEAAgMAAAAAAAAAAAAAAAABABEQITH/2gAIAQMBAT8QBVM1Hrj/xAAXEQEBAQEAAAAAAAAAAAAAAAABEDFB/9oACAECAQE/EE07BhP/xAAcEAEBAAMAAwEAAAAAAAAAAAABEQAhMVFhcaH/2gAIAQEAAT8QUhkEVKfHjA10KQp0jMUp9wY6EihV16w0n7o+2sU+sPC/mXadbXuBNStjW5gkDgZ//9k=","width":50,"height":50,"src":"/static/d37e68c1bb8fcdb464c251c9a15e7dbe/d2d31/dhivya.jpg","srcSet":"/static/d37e68c1bb8fcdb464c251c9a15e7dbe/d2d31/dhivya.jpg 1x,\n/static/d37e68c1bb8fcdb464c251c9a15e7dbe/0b804/dhivya.jpg 1.5x,\n/static/d37e68c1bb8fcdb464c251c9a15e7dbe/753c3/dhivya.jpg 2x,\n/static/d37e68c1bb8fcdb464c251c9a15e7dbe/31ca8/dhivya.jpg 3x"}}},"headerImage":null},"html":"Test data management using SHIFT
\nThe test cases, scenarios, and strategies are maintained in SHIFT, our internal (and soon to be open-sourced) test case management system. Our QAs make use of a test data tool which is integrated to SHIFT. The 100+ types of test data required by the teams are generated using automation suites that run in Jenkins. This test data is then queued in SHIFT and as a result increased the productivity of our QA team by reducing the effort for manual test data creation. Our director of QA has briefed this here https://auto1.tech/faster-automation-tests/.
\nSeamless Integrations
\nOur CI system, Jenkins, is wired to run these tests against our various QA environments. The integration of SHIFT with Slack, Jenkins, and Jira has helped the teams immensely. The status of Jenkins jobs/sanity suites is notified in Slack. The time taken to debug the issue is hugely reduced since SHIFT displays a precise list of failed tests including the screenshot reference, which helps me to quickly understand the exact point of failure. Once I have identified the issue, I fix it and test it locally, then push the branch to our repository. The process is set in such a way that merge can be done only after review, which gives me more confidence and validation for the proposed solution. This helps me address the issues faster and get the system up and running. This is indeed why the QA team at AUTO1 is one of the key pillars of the Engineering team.
\nMy automation testing journey at AUTO1 could not have been this impactful if not for this perfect blend of the best tech stacks and resources, and most importantly, my amazing team. My learning on the job has been very effective and hands-on, and it makes each day filled with new challenges and new learning opportunities that help my work create a real impact.
\nFrom the very many opportunities that came my way and the deep learning curve it has presented me with, I now realize that - the journey has just begun!
","fields":{"slug":"/qa-engineer-journey/","tags":["auto1","engineering","testing","QA","social"]}}},{"node":{"id":"0e79f71e-bac7-5c18-b5ae-ae1cec933364","frontmatter":{"category":"QA","title":"Efficient testing - Test Strategy","date":"2021-10-08","summary":"The first article in a series of articles focusing on efficient testing.","thumbnail":null,"authorName":"Michal Chytroszek","authorDescription":"Michal is the QA lead of the Car Journey associated teams.","authorAvatar":{"relativePath":"pages/efficient-testing-1/Michal.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAACE4AAAhOAFFljFgAAAEyklEQVQ4yx2Qe0xTdxzFr9Dee3/30dpCRQqsQofyEEQpT8v6ABwgUPoASpUiMigKHeVReZTy0BYEh6TVtohKsbCNqAiFKPhgY3GYqAPMhkaWzEyXbG5z2ZZliX8sY5clJ998//mcnHMgg83Tah28e+2k7Uxb36Uxy/B4w1mP4bSr8WRvtjQ5I57X47R3uYY1avkxfZX+hL5CKXbUZpbLJVVHiiDLN/+euvfDl4/mHj24dnXh4f1Xv0y/+Htw6fehlbfywxWi2OCx5VfO1bfuJ3/VWFy9o580G/WL5ysHLnpu3ZyFWj//o8O7dnv5q/kHS947M3efPn/8+rfTX/w0+PXG8c7+tChu0+BQdfegXG/pu70+v/6yratjskfjmrhxzjUCGRfedCz8emlqbuaze8vfrV9f+VZfU5knTTZYBuptngOJkbWGWm1pcUpyYk2T0dRmqNOVTnRrHOesDVUlkME+1uC62ugY7xy91uWezN2/dxcORTKgZC5Q5OWLBRFFMmleekJq3E5VrrDq6OHGD6tnzpW2dylOlL4HWV9unH650ff9P2d+3miafiiPZjsVPLcywFPo156zI3k3L1McJ0qKEQkiNIdy7cPOZrNJeZinqhRrVRlQ48Wp5pFZ08h0z/iMuX/AKg/3loXNargLWn/PIX62MFp+MEEiTk3dHaKV729oNqoz99QXBLVo4oqyEqDe8VvOqzfdc/cvTM17LjsvHxEURm01p3F0e4haEV99ME2pSpfnC3X52ztNOaOrb0ZbBet96NDxhPJDBdDd1bVnKzf+fHZlbu3FWH+HVxM2URQ6VcQbzd3mLQmpFocnJsY0lPMrlL4HFUzPpOO6VeIyZelbzCp9N9Qy/Wzx4eKPT9w3nzx3zy/OdmtsGQH6ff7mRMZIQYBKEiMWhEglHJmCa+zQdtts5o/sirreYrNDVtsBtc6s9dx5ar+zapl9fGrp9YBzyKMK7s0MtmWSLYWJkXsTynRHjfYLdYOjdbbrOqu73GwvrOss6XTJDFZI//GDdu9y/8K62btSc35SmJ5dKeTqkvzUsUQEf0dQME+Up9G2n6/q81RYhss6HdqWM7Jqk6p9KFdvgeo/Xapz3pDK1Pxd0SwCp/v44BgRzsEJBN6yxRem03GaDxOGRPkardlR3NhXXG/NLm/KMpyVVrRBxomlhP1SGIIQmi8GAIskEASFaHQUhnGAogBlEgSB+PBC+craroJjZpnOlK4+Li5rFhbqoIr+KwH+fjiG4BhKAJRFYAwcENRPXRzdFEBJgHJYzKwSXf7RxpxSvViuFRYcScophuLfVwZu8wOIL0BpBEZn4YBNiYExCUASgEGgGIqQGMLAkLSMvJziDyT56tQDBXHvZUclSaFIsYIbGECgvghCwwHMJgCbRCmxCJRFUjCgYCaOMgEtIipGmJEnSJVEx6cEvhO2LZgHCVJEASyCxGAEoQMU3mQIqie6dfMBJE61RthMwMThwO3bY+NTouME/KgYNjeY5HCg0CD/QDZB4hRJg2E65bIZEv9f1BBg05RNYiwCYbMZEbH7dkbH8sIjWIFBwI8NZWuqOAyEScIYSqPBvgigAwBjGExNCFA6RdLoNCoIi4EyGei7UbFhuyJDQncw/P0o+/8A3meRLxNk0Z4AAAAASUVORK5CYII=","width":50,"height":50,"src":"/static/52659ecfce9ed71e567bddef5d9c1b15/45876/Michal.png","srcSet":"/static/52659ecfce9ed71e567bddef5d9c1b15/45876/Michal.png 1x,\n/static/52659ecfce9ed71e567bddef5d9c1b15/eb85b/Michal.png 1.5x,\n/static/52659ecfce9ed71e567bddef5d9c1b15/4f71c/Michal.png 2x,\n/static/52659ecfce9ed71e567bddef5d9c1b15/9ec3e/Michal.png 3x"}}},"headerImage":null},"html":"Hi everyone! This article is the first in a series of AUTO1 testing-related posts, I hope you’ll find the content interesting enough to return for the rest in the series :) To start off, let’s ask one of the most important questions:
\nWhat is a Test Strategy?
\nIn AUTO1 the test strategy is a high level, short and concise document which defines:
\n- \n
- \n
Scope of the testing
\n \n - \n
Risks - areas that potentially can be broken
\n \n - \n
Automation strategy
\n \n - \n
Required test data
\n \n - \n
Security
\n \n
Why do we create a Test Strategy?
\nIn agile and fast changing environments we usually do not spend that much time on test documentation as in waterfall. However, even in such cases we cannot underestimate the importance of a test strategy. When you are creating a strategy it becomes your guide, a channel for your thoughts - you have a space to stop for a moment to try to understand what should be verified and do it at a very early stage. It is much easier to go back to the feature if you already wrote down your thoughts and you have some kind of summary.
\nHow do we create a Test Strategy?
\nLet’s go through our thinking process, step by step.
\nScope of the testing
\nStart with defining what exactly has to be tested. You can ask yourself a few questions:
\n- \n
- \n
Who is going to use the feature?
\n \n - \n
What apps will be changed/created?
\n \n - \n
On what devices will it be used?
\n \n - \n
What types of testing will be needed?
\n \n
It allows you to better understand the product you are going to test and the environment setup you’ll need.
\nRisks
\nNow think about all the things that can go wrong. Here the questions will be:
\n- \n
- \n
What apps/areas can be impacted by our change?
\n \n - \n
Are we modifying dependencies of another system?
\n \n - \n
Is my domain knowledge enough to understand a feature?
\n \n - \n
Are there any missing prerequisites which can block testing?
\n \n - \n
Are changes going to be compatible with existing solutions?
\n \n
Answering these questions will help us create a plan in order to mitigate all risks. It can be planning an onboarding session with someone who has knowledge we need, investigating regression testing possibilities for apps or areas in danger or setting up a plan on how to get blockers done.
\nAutomation strategy
\nAutomation is an important part of our testing process. We think about the approach on the test strategy stage. We create our tests on an UI and API level. When we’ll have the technical specifications, which include details of implementation, then it is time to plan what and when will be automated. We ask questions such as:
\n- \n
- \n
Where is the specific logic placed? Is it frontend or backend?
\n \n - \n
Do we have existing automation in this area? If yes, do we have to adjust it?
\n \n - \n
Do we have all prerequisites needed to start writing automation?
\n \n
Required test data
\nAt AUTO1 we have a lot of connections between several areas. It created dependencies which have to be covered in test data preparation. We have to be sure we know the preconditions and steps required to arrange the system before starting testing. Here we can specify if we need additional onboarding or support from other teams.
\nHaving answers to the above questions means we are prepared to plan more detailed test cases and we can use our strategy as some kind of a cheatsheet. However, do not blindly follow what you’ve planned - requirements can change. Review your strategy from time to time, to ensure that it is not outdated.
\nSummary
\nTest strategy shouldn’t take much time but it is important for the next step - creating a test plan. It can be a great guide which will help you to structure your initial ideas and be prepared for upcoming changes faster.
","fields":{"slug":"/efficient-testing-1/","tags":["auto1","engineering","testing","QA"]}}},{"node":{"id":"7efdac28-b97f-53e0-adf8-48d022f3eecc","frontmatter":{"category":"Coding","title":"Integration Test Speedup with Spring","date":"2019-03-11","summary":"Here is what we found on speeding up your integration tests in spring based applications.","thumbnail":null,"authorName":"Boris Faniuk","authorDescription":"Boris is a former member of Engineering of AUTO1 Group.","authorAvatar":{"relativePath":"pages/integration-test-speedup/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/integration-test-speedup/header.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAPABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAABQABAv/EABYBAQEBAAAAAAAAAAAAAAAAAAIAAf/aAAwDAQACEAMQAAABJ6Uw4FJSv//EABsQAAICAwEAAAAAAAAAAAAAAAECAAMEFCEx/9oACAEBAAEFAl9tuawzH6WUPNYT/8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPwE//8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQIBAT8BJ//EABsQAAEEAwAAAAAAAAAAAAAAAAABEBIhIjFR/9oACAEBAAY/AjJo9SizZ//EABwQAQEAAgIDAAAAAAAAAAAAAAERAHEQMUFR0f/aAAgBAQABPyFMdusqfAgeuBQwbbGHwF1jT4z/2gAMAwEAAgADAAAAELsf/8QAFhEBAQEAAAAAAAAAAAAAAAAAARBB/9oACAEDAQE/EA2f/8QAFxEAAwEAAAAAAAAAAAAAAAAAAAERIf/aAAgBAgEBPxDSpD//xAAbEAEBAQADAQEAAAAAAAAAAAABEQAhMXFBYf/aAAgBAQABPxAMiBZ241OQL4wZbyughIofpPEuZhdFS61EeZ//2Q==","width":1024,"height":768,"src":"/static/9f60dcc74069c89c91db86dc33946167/9f594/header.jpg","srcSet":"/static/9f60dcc74069c89c91db86dc33946167/9f594/header.jpg 1x"}}}},"html":"Problem statement
\nOver time we realized that our microservices built on top of Spring framework take tons of time on the integration test step. Every single PR runs all tests and this takes 10-20 minutes for some services. To decrease PR build time we decided to research why this happens.\nOne of the problems that this research highlighted is ramping up heavy spring context several times during execution of all module’s tests.
\nAs a proof of concept we have taken 3 modules and cleaned up their tests to achieve better integration test performance.
\nThe results are pretty much self-explanatory:
\n- \n
- Module A now takes 3 minutes to test versus 11 before \n
- Module B now takes 3 minutes to test versus 8 before \n
- Module C now takes 4 minutes to test versus 12 before \n
Below I list all factors that produce many spring contexts and how we managed to overcome them.
\nSpring features to be avoided
\n@ActiveProfiles and @ContextConfiguration with different attributes
\nOur spring-based microservices actively use spring profiles and configuration classes.\nMany components from common libraries are hidden under profiles and configuration classes to not be active for every single module / environment without need. So, integration test authors define profiles and configuration classes that should be activated for their tests using these annotations:
\n- \n
@ActiveProfiles({\"integration-test\", \"pg-test\", \"mongo-test-two-nodes\"})\n@ContextConfiguration(classes = {Application.class, AwsTestConfiguration.class})\n
Here mongo-test-two-nodes activates special mode for mongo based services to operate with two separate mongo instances. As not all tests need this, the profile is turned off by default and only those tests that need this behaviour would explicitly turn it on. But this, unfortunately, leads to multiple spring contexts being created.
\nThe same happens with AwsTestConfiguration that deploys special test-component to mock AWS calls.
\nThe solution was to make all tests use the same set of active profiles and context configuration classes for every single integration test.\nThe drawback is that some components are deployed, even for the tests that do not need it. But deploying single component takes O(1) time comparing with the whole spring context.
\nAnother point to keep in mind is that order of profiles (configuration classes) matters.\n@ActiveProfiles({\"integration-test\", \"pg-test\"})\n@ActiveProfiles({\"pg-test\", \"integration-test\"})
Two settings mentioned above result in 2 different spring contexts.
\n@SpyBean and @MockBean
\nOne interesting feature in spring-boot-test is mocking / spying real spring beans.\nThis was mostly used when we don’t care what some method does in test but we need to check that it was called by parent components.\nUnfortunately this also makes test to create separate spring context.\nWhat we’ve done here is replacing @SpyBean annotation with configuration class that wraps real spring bean into Mockito.spy(..):
@Configuration\npublic class SomeComponentSpyFactory {\n\t@Autowired\n\tprivate SomeComponent someComponent;\n\n\t@Bean\n\t@Primary\n\tpublic SomeComponent someComponentSpy() {\n \t\treturn Mockito.spy(someComponent);\n\t}\n}Another option is to use the combination of @Autowired and @Spy annotations.
\t@InjectMocks\n\t@Autowired\n\tprivate ParentComponent parentComponent;\n\n\t@Spy\n\t@Autowired\n\tprivate ChildComponent childComponent;However, the drawback of the last solution is that, parentComponent needs to have a setter for childComponent which is not good in terms of encapsulation. But if you don’t use autowiring on constructor level it is fine. Anyway, one can choose whatever fits the situation better.
\n@DirtiesContext
\nOne of the services used this annotation to clear the caches between different tests.\nClearing a cache is really a problem (see section below), but this solution comes to separate spring context per test class or even test method (depending on annotation parameters). Context per method was actually the case for a few test classes. Instead in this modules we added methods to clear the caches programmatically without deploying the whole new context.\nOne can use java reflection API instead of adding special production-level methods, again whatever fits you better.
\nChallenge: test execution order and cleaning collections (caches)
\nAfter all this “multiple context” factors are gone one may face (and we did) a challenge to fix or re-organize some tests. Before above enhancements were done all tests were running in separate spring contexts, therefore used separate instances of every component and therefore those problems could be hidden.
\nAs an example some tests in mongo-based service were testing getAll()-like operations. Those tests were running functions that fetched all records from some collections. This was not a problem as embedded mongo DB was different instance. After all tests started to exploit the same spring context some of them started to fail intermittently. The failures were intermittent because test execution order is different. One service may break another, but not vice versa.\nSo, to overcome this, we added missing @Before operations that cleaned those collections.
The rule that I would suggest here is that every test should clean up whatever it needs to be clean and not rely on fairness of other tests. That was a problem in our case. Some tests had @After operations that cleaned up, but not everything produced by the test was cleaned up. This caused intermittent failures of other tests.
Tools we created to reduce test execution time
\nSo, now after we know all this, we created two tools (and may create more in future) that help to identify those problems.
\nJenkins PR pipeline step
\nAfter every PR commit we run one additional step that analyzes test execution log.\nIf more than one spring context is created this leaves some traces and a notification is sent to the author of the commit via slack
\nAs an enhancement, the jenkins jobs could be modified to fail the build if the number of spring contexts is greater than a certain threshold. The threshold could start with a high number and can gradually be reduced as the team cleans up the existing tests by adopting the aforementioned practices/fixes
\nIntelliJ IDEA plugin
\nWe wrote an IntelliJ plugin to identify and solve the aforementioned problems. There will be a separate blog post about this plugin coming very soon.
","fields":{"slug":"/integration-test-speedup/","tags":["auto1","engineering","spring","java","testing"]}}},{"node":{"id":"458b814a-9266-529b-a538-b64d05af8cc6","frontmatter":{"category":"QA","title":"How to O(1) your test preparation","date":"2019-01-29","summary":"Are you sure your tests are fast enough?","thumbnail":null,"authorName":"Alexander Gyulai","authorDescription":"Head of Quality Assurance at AUTO1 Group","authorAvatar":{"relativePath":"pages/faster-automation-tests/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/faster-automation-tests/header.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAIAAACHqfpvAAAACXBIWXMAAAsSAAALEgHS3X78AAABtklEQVQY0wGrAVT+AAACBAAEBwAECAQECA0ECRoECCwFCEEFCVYECGsECHkECYYECZUGDKIKEKEMEaMLEqcQHK4aJKo5PjghPwBTAwdiAgRvAQOBAQONAAORAQWOAweFBAmCCA18Cw9zDhFlEhRMEhI4ERdGHC5XOlZyXHx9hqiGosqfkLcAXQEEUwIFSgkNNgsQJA0SHxMWGBMVERUVCBMSARAPAA8SAQ0VAAsjHy1Rd36i0NTk9Pv//f3+/8zj14CnAAcSFgUTFgcQEAoQDgsODAYICAUGCQUFCwQIEwQIGwIJLBAbRz9Md5CevNje7f/q8PvT4c6Yu21CexENPgAIBQEHAwMBAAUAAAgABBQCCRkDDB0CDB8LEy8tN1xub465qb/uwdb3r8zdi7i4cKZ5VYsgJlkABSkBAhYAAAIQAAUTAwobBxAgBA4ZCBAfISY4WFtznZau0rTI76rD5YGoxlaSkzyAVS9xGiRcABM+AQklBQkXAQQLAAYOGAsVHw4ZIBUaJ0dDWJGOoM3Azty0xtGIqbZXiaA7fY0qdUwkZREbUgAQPQAGJAMEFQMGEgACBwAAAMagbU9WOO54AAAAAElFTkSuQmCC","width":1280,"height":445,"src":"/static/34ed0cae6856ac71c7fea1ac46b6a5a9/26421/header.png","srcSet":"/static/34ed0cae6856ac71c7fea1ac46b6a5a9/26421/header.png 1x"}}}},"html":"(*) If your career as a QA - like the author’s - started without a computer science degree, you will hopefully love\nto hear and learn about the different theories and notations of runtime analysis.\nCheck wikipedia’s page about O-notation here
\nSpeed… Did you ever think how you can make your tests faster and more reliable? If you're anything like me or my colleagues at AUTO1,\nyou face this challenge all the time. Here at AUTO1 we are constantly looking for new ways to improve our tools and\nto make testing fast and efficient. In this article, I’m going to take test suite that takes ~70 min and reduce the execution time to 1 min!!!\nInteresting, right? :)
\nIntro
\nLet’s imagine we are testing some Web Shop Scenario. We need to test that users can do some actions with items they purchased\n(submit feedback, return, submit claim, etc.). And imagine that in this case we have exactly 20 scenarios that go like this:
\nSteps:\n- Create item, put item in certain state\n- Create user\n- Create user delivery address\n- Create user billing address\n- Create user payment details\n- Sell item to user\n- User does something with an itemBack in times…
\nLet’s look at the timeline of our test
\nWe significantly increased the speed. From ~73 to ~8 minutes. This already looks like a game changer! But can we do more? Spoiler: Yes we can!
\nRemoving dependencies
\nIf we check our test scenario again, we can clearly see that the actual validation of our test scenario happens only\nafter the third minute! And everything we have up until then it’s just creating preconditions.\nWhat we actually want to test is: “User can do certain action with item purchased on our platform”.\nWe should not care HOW we got this item, we should have another test, testing that user should be able to purchase.\nThis just creates dependency for us that makes our tests:
\n- \n
- Slower \n
- Less stable. It becomes less stable because we have to go through multiple steps (in some cases maybe through multiple\napplications) with creating item, creating user, adding user info and so on. If something will fail in the middle (which can happen easily)\nit doesn’t really mean that user couldn't leave a feedback on the item. \n
P.S. While writing this article I tried to re-create all these conditions and I didn’t have any successful build.\nAnd it was always something related to other dependencies.
\nSolution? There are many possible answers, the one I chose for this example is to move all logic with test data preparation to the API layer.\nLet’s try this and review our timeline again
\n
Amazing, this is how our test looks now! two minutes just gone.
\n And this is how our suite looks now comparing to other runs.\n Having test preparation using API is a great step here, but does it scale from 20 tests to 1000 without having side-effects?\nAt the end, your test data preparation might look like this: Assuming you have around 40 calls with a reasonable fan-out for every scenario and you usually run tests in multiple threads,\nyou might involuntary start a load test there. Aaaaand it’s time to look for a solution again! What if we have our test data waiting for us somewhere? Wouldn’t be amazing?!\nOur solution for this problem was to create a Web Service that would keep the data in a queue for us.\nIt provides couple of huge benefits for us: Let’s look at our timeline again. This is how our test scenario changed: We barely can see it on our timeline :) Stay tuned! We will open-source it soon! In AUTO1 Group, every day we are looking for smarter ways to do software development and testing.\nAnd the main point in this article is that you should never stop innovating and looking for new solutions to existing problems.\nSomething that works for us, is not necessarily going to work for you,\nbut maybe you will ask yourself: “Where is our next level? Can we do something better?”![\"machingun\"](\"/static/machingun-f6e9bbc355090a0e8c6db2e891dfb64a.gif\" "\\"Suitemachingun\\"")
Introducing Q-Service
\n\n
\n\n
\nHow does it work?
\nSummary
\n